2025-06-29 02:38:53,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-905b9981]) (logid:a9113453) No inactive management server node found 2025-06-29 02:38:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-905b9981]) (logid:a9113453) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:38:53,388 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-adb95e2e]) (logid:6ec4eea2) HA health check task is running... 2025-06-29 02:38:54,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16b51170]) (logid:f5aec98f) No inactive management server node found 2025-06-29 02:38:54,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16b51170]) (logid:f5aec98f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:38:55,031 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7819: Seq 7-7819: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:38:55,041 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7819 2025-06-29 02:38:56,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 02:38:56,215 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Seq 2-3048092522799696588: Routed from 90520738888109 2025-06-29 02:38:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a1f857c]) (logid:e79afb26) No inactive management server node found 2025-06-29 02:38:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a1f857c]) (logid:e79afb26) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:38:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d2c27f5e]) (logid:72fc1e89) Enabled non-edge zones available for scan: 1 2025-06-29 02:38:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d2c27f5e]) (logid:72fc1e89) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:38:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d2c27f5e]) (logid:72fc1e89) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:38:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f484862]) (logid:9c4133a3) Enabled non-edge zones available for scan: 1 2025-06-29 02:38:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f484862]) (logid:9c4133a3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:38:57,295 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:38:57,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:38:57,299 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) Seq 2-3048092522799696589: Routed from 90520738888109 2025-06-29 02:38:57,353 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) Seq 1-3048092522799696589: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:38:57,389 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a945dd82]) (logid:29deb962) HA health check task is running... 2025-06-29 02:38:57,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:38:57,401 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) Seq 2-1612007191621929858: Routed from 90520738888109 2025-06-29 02:38:57,452 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-cab6292d]) (logid:1c89c1a7) Seq 2-1612007191621929858: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:38:57,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10dfaecb]) (logid:6d4bfba3) No inactive management server node found 2025-06-29 02:38:57,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10dfaecb]) (logid:6d4bfba3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:38:57,853 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-d7e5060a]) (logid:b3876c6e) AutoScaling Monitor is running... 2025-06-29 02:38:57,946 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-67aaa553]) (logid:1f642d5b) Starting to remove all volume_stats rows older than [Sat Jun 28 14:38:57 CEST 2025]. 2025-06-29 02:38:57,947 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-67aaa553]) (logid:1f642d5b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:38:57 CEST 2025]. 2025-06-29 02:38:59,240 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Seq 1-3048092522799696588: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 02:38:59,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 02:38:59,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 02:38:59,271 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Seq 2-1612007191621929859: Routed from 90520738888109 2025-06-29 02:38:59,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1680ce6]) (logid:e9774216) No inactive management server node found 2025-06-29 02:38:59,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1680ce6]) (logid:e9774216) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:00,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af1a316b]) (logid:77ffce9a) No inactive management server node found 2025-06-29 02:39:00,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af1a316b]) (logid:77ffce9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:01,390 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-bea09ac9]) (logid:16fd3c3f) HA health check task is running... 2025-06-29 02:39:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e39234]) (logid:fb2de4c9) No inactive management server node found 2025-06-29 02:39:02,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e39234]) (logid:fb2de4c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:03,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cce2daa0]) (logid:bbe51181) No inactive management server node found 2025-06-29 02:39:03,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cce2daa0]) (logid:bbe51181) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:04,366 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Seq 2-1612007191621929859: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 02:39:04,366 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-30ea3fc0]) (logid:2a99cc35) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 02:39:05,031 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7820: Seq 7-7820: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:05,052 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7820 2025-06-29 02:39:05,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b48a9963]) (logid:a1af09ca) No inactive management server node found 2025-06-29 02:39:05,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b48a9963]) (logid:a1af09ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:05,392 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2e574160]) (logid:785d6b94) HA health check task is running... 2025-06-29 02:39:06,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec14b7a3]) (logid:205bb4a0) No inactive management server node found 2025-06-29 02:39:06,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec14b7a3]) (logid:205bb4a0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-285aed7a]) (logid:b7f1efad) No inactive management server node found 2025-06-29 02:39:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-285aed7a]) (logid:b7f1efad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:09,393 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2527a6d3]) (logid:9e2e4dab) HA health check task is running... 2025-06-29 02:39:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45102c0f]) (logid:5d36e152) No inactive management server node found 2025-06-29 02:39:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45102c0f]) (logid:5d36e152) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:10,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7821: Seq 7-7821: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:10,048 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7821 2025-06-29 02:39:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1115c4c9]) (logid:5c22fb05) Found 0 routers to update status. 2025-06-29 02:39:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1115c4c9]) (logid:5c22fb05) Found 0 VPC's to update Redundant State. 2025-06-29 02:39:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1115c4c9]) (logid:5c22fb05) Found 0 networks to update RvR status. 2025-06-29 02:39:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Running Capacity Checker ... 2025-06-29 02:39:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) recalculating system capacity 2025-06-29 02:39:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Executing cpu/ram capacity update 2025-06-29 02:39:11,069 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:11,070 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:11,076 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 02:39:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 02:39:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 02:39:11,104 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:11,105 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:11,109 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:11,111 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 02:39:11,111 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 02:39:11,118 DEBUG [c.c.c.CapacityManagerImpl] (pool-304-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 02:39:11,126 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Done executing cpu/ram capacity update 2025-06-29 02:39:11,126 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Executing storage capacity update 2025-06-29 02:39:11,132 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:39:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 02:39:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 02:39:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 02:39:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:39:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 02:39:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 02:39:11,141 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 02:39:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:39:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 02:39:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 02:39:11,154 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 02:39:11,162 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:39:11,163 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 02:39:11,163 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 02:39:11,167 DEBUG [c.c.s.StorageManagerImpl] (pool-305-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 02:39:11,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Done executing storage capacity update 2025-06-29 02:39:11,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Executing capacity updates for public ip and Vlans 2025-06-29 02:39:11,181 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Done capacity updates for public ip and Vlans 2025-06-29 02:39:11,181 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Executing capacity updates for private ip 2025-06-29 02:39:11,185 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Done executing capacity updates for private ip 2025-06-29 02:39:11,185 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Done recalculating system capacity 2025-06-29 02:39:11,212 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-cdb3c1f9]) (logid:513aa598) Done running Capacity Checker ... 2025-06-29 02:39:11,257 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b5cd409a]) (logid:637950d9) Checking if any host reservation can be released ... 2025-06-29 02:39:11,262 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b5cd409a]) (logid:637950d9) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:11,265 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b5cd409a]) (logid:637950d9) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:11,266 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b5cd409a]) (logid:637950d9) Done running HostReservationReleaseChecker ... 2025-06-29 02:39:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ea8f568]) (logid:58749683) No inactive management server node found 2025-06-29 02:39:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ea8f568]) (logid:58749683) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:12,779 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-bac3bef8]) (logid:a3d142bf) Starting to remove all vm_stats rows older than [Sat Jun 28 14:39:12 CEST 2025]. 2025-06-29 02:39:12,794 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-bac3bef8]) (logid:a3d142bf) Removed a total of [0] vm_stats rows older than [Sat Jun 28 14:39:12 CEST 2025]. 2025-06-29 02:39:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff3db7f9]) (logid:498669b3) No inactive management server node found 2025-06-29 02:39:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff3db7f9]) (logid:498669b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:13,394 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-77e3072a]) (logid:76bf1897) HA health check task is running... 2025-06-29 02:39:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc6893e]) (logid:830f9d74) No inactive management server node found 2025-06-29 02:39:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc6893e]) (logid:830f9d74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:15,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f5cc9e9]) (logid:a4ac4a5d) No inactive management server node found 2025-06-29 02:39:15,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f5cc9e9]) (logid:a4ac4a5d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:17,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37d9e986]) (logid:c75dbe2d) No inactive management server node found 2025-06-29 02:39:17,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37d9e986]) (logid:c75dbe2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:17,395 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5eb627fb]) (logid:d290df3a) HA health check task is running... 2025-06-29 02:39:18,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04ff283e]) (logid:8c2638e6) No inactive management server node found 2025-06-29 02:39:18,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04ff283e]) (logid:8c2638e6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:20,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7822: Seq 7-7822: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:20,059 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7822 2025-06-29 02:39:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2e607d]) (logid:e0243a4e) No inactive management server node found 2025-06-29 02:39:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2e607d]) (logid:e0243a4e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:21,397 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e7b75c54]) (logid:14e8fb81) HA health check task is running... 2025-06-29 02:39:21,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-111c6436]) (logid:bfc9454f) No inactive management server node found 2025-06-29 02:39:21,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-111c6436]) (logid:bfc9454f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:23,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae1ec3eb]) (logid:18ce7744) No inactive management server node found 2025-06-29 02:39:23,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae1ec3eb]) (logid:18ce7744) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:23,380 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) StorageCollector is running... 2025-06-29 02:39:23,387 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:39:23,387 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:39:23,387 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:39:23,387 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-6717118844223095456: Routed from 90520738888109 2025-06-29 02:39:23,440 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 4-6717118844223095456: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:39:23,443 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:39:23,443 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:39:23,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:39:23,445 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-3048092522799696590: Routed from 90520738888109 2025-06-29 02:39:23,493 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 1-3048092522799696590: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:39:23,505 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:39:23,505 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:39:23,505 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:39:23,506 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-1612007191621929860: Routed from 90520738888109 2025-06-29 02:39:23,517 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-1612007191621929860: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:39:23,530 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:39:23,530 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:39:23,530 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:39:23,532 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-3048092522799696591: Routed from 90520738888109 2025-06-29 02:39:24,582 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 1-3048092522799696591: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:39:24,583 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:39:24,583 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:39:24,583 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:39:24,584 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-1612007191621929861: Routed from 90520738888109 2025-06-29 02:39:24,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4129fcb]) (logid:de02a076) No inactive management server node found 2025-06-29 02:39:24,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4129fcb]) (logid:de02a076) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:25,398 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-342583c0]) (logid:fa5af0de) HA health check task is running... 2025-06-29 02:39:25,647 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-67d033d5]) (logid:44843dfa) Seq 2-1612007191621929861: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:39:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f2f02ac]) (logid:01d9f9fc) No inactive management server node found 2025-06-29 02:39:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f2f02ac]) (logid:01d9f9fc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:26,769 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1a99e27a]) (logid:8140b35f) Enabled non-edge zones available for scan: 1 2025-06-29 02:39:26,774 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1a99e27a]) (logid:8140b35f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:39:26,776 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1a99e27a]) (logid:8140b35f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:39:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6528b397]) (logid:e5d5efe0) Enabled non-edge zones available for scan: 1 2025-06-29 02:39:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6528b397]) (logid:e5d5efe0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:39:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49ac18e9]) (logid:3913e998) No inactive management server node found 2025-06-29 02:39:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49ac18e9]) (logid:3913e998) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:29,132 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:39:29,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:39:29,134 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) Seq 2-3048092522799696592: Routed from 90520738888109 2025-06-29 02:39:29,137 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) Seq 1-3048092522799696592: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:39:29,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:39:29,138 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) Seq 2-1612007191621929862: Routed from 90520738888109 2025-06-29 02:39:29,141 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-60df66a5]) (logid:0a6eb8ec) Seq 2-1612007191621929862: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:39:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1464f6b1]) (logid:b081bf64) No inactive management server node found 2025-06-29 02:39:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1464f6b1]) (logid:b081bf64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:29,399 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9ea9836a]) (logid:7db769d8) HA health check task is running... 2025-06-29 02:39:30,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7823: Seq 7-7823: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:30,042 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7823 2025-06-29 02:39:30,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9bce1a8]) (logid:e2232f54) No inactive management server node found 2025-06-29 02:39:30,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9bce1a8]) (logid:e2232f54) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b330346]) (logid:615edec5) No inactive management server node found 2025-06-29 02:39:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b330346]) (logid:615edec5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:33,400 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-504c378b]) (logid:3be82409) HA health check task is running... 2025-06-29 02:39:33,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6907129e]) (logid:f19ab707) No inactive management server node found 2025-06-29 02:39:33,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6907129e]) (logid:f19ab707) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:39:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:39:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9964028b]) (logid:1d0e0017) No inactive management server node found 2025-06-29 02:39:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9964028b]) (logid:1d0e0017) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-934bd908]) (logid:80ea7615) No inactive management server node found 2025-06-29 02:39:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-934bd908]) (logid:80ea7615) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:37,402 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a129f6b0]) (logid:d0f36984) HA health check task is running... 2025-06-29 02:39:38,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3e115af]) (logid:6cd47176) No inactive management server node found 2025-06-29 02:39:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3e115af]) (logid:6cd47176) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:39,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e540b6e7]) (logid:8de3e2e5) No inactive management server node found 2025-06-29 02:39:39,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e540b6e7]) (logid:8de3e2e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:39:40,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7824: Seq 7-7824: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:40,043 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7824 2025-06-29 02:39:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-077f0f7b]) (logid:549da442) Found 0 routers to update status. 2025-06-29 02:39:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-077f0f7b]) (logid:549da442) Found 0 VPC's to update Redundant State. 2025-06-29 02:39:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-077f0f7b]) (logid:549da442) Found 0 networks to update RvR status. 2025-06-29 02:39:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-fbd60d31]) (logid:621106c8) DbCollector is running... 2025-06-29 02:39:41,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bba381b4]) (logid:4c871cd0) No inactive management server node found 2025-06-29 02:39:41,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bba381b4]) (logid:4c871cd0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:41,403 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9ccc9809]) (logid:20c87178) HA health check task is running... 2025-06-29 02:39:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) ManagementServerCollector is running... 2025-06-29 02:39:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) DB properties were already loaded 2025-06-29 02:39:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:39:42,168 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Execution of process [4061688] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:39:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Exit value of process [4061688] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:39:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Process [4061688] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:39:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) usage install: null 2025-06-29 02:39:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Sessions active: 0 2025-06-29 02:39:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:39:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061691] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:39:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:39:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061696] for command [/bin/bash -c uname -r ]. 2025-06-29 02:39:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:39:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061697] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:39:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) system memory from /proc: 4887515136 2025-06-29 02:39:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:39:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061702] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:39:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) free memory from /proc: 514957312 2025-06-29 02:39:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:39:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061707] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:39:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) used memory from /proc: 753036 2025-06-29 02:39:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:39:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061708] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:39:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:39:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061709] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:39:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:39:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061710] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:39:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:39:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061717] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:39:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:39:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Successfully executed process [4061718] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail buffersdirect.count, 120 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail buffersdirect.used, 657986 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail buffersmapped.count, 7 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail buffersmapped.used, 798602 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail gcPS-Scavenge.count, 472 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail gcPS-Scavenge.time, 3472 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail jvmuptime, 41522318 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memoryheap.committed, 348127232 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memoryheap.init, 77594624 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memoryheap.usage, 0.10008240584258489 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:39:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorynon-heap.max, -1 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorynon-heap.usage, -2.2007288E8 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorynon-heap.used, 220072880 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17512829912023462 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.PS-Eden-Space.usage, 0.024640904452505427 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.PS-Old-Gen.usage, 0.11558773525465718 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorypools.PS-Survivor-Space.usage, 0.8093826293945312 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorytotal.committed, 572981248 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorytotal.init, 85262336 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorytotal.max, 1908932607 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail memorytotal.used, 411188152 2025-06-29 02:39:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail threadsdeadlocks, [] 2025-06-29 02:39:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail threadsnew.count, 0 2025-06-29 02:39:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:39:42,245 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-ac55349a]) (logid:2e0ca815) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:39:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348127232,"freeJvmMemoryBytes":157831072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":410367248,"totalCommitted":572981248,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191061352,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514957312,"systemMemoryUsed":753036,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.38,0.26],"systemCyclesUsage":[66359828,53800632,4587343485],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:39:42,245 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1383, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:39:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348127232,"freeJvmMemoryBytes":157831072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":410367248,"totalCommitted":572981248,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191061352,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514957312,"systemMemoryUsed":753036,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.38,0.26],"systemCyclesUsage":[66359828,53800632,4587343485],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:39:42,245 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:39:42,250 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-309:[ctx-683c414d]) (logid:3006d0f7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:39:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348127232,"freeJvmMemoryBytes":157831072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":410367248,"totalCommitted":572981248,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191061352,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514957312,"systemMemoryUsed":753036,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.38,0.26],"systemCyclesUsage":[66359828,53800632,4587343485],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:39:42,250 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:39:42,250 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1383, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:39:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348127232,"freeJvmMemoryBytes":157831072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":410367248,"totalCommitted":572981248,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191061352,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514957312,"systemMemoryUsed":753036,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.38,0.26],"systemCyclesUsage":[66359828,53800632,4587343485],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:39:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c806b2b1]) (logid:74442026) No inactive management server node found 2025-06-29 02:39:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c806b2b1]) (logid:74442026) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f424ca17]) (logid:d8c3ffe8) No inactive management server node found 2025-06-29 02:39:44,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f424ca17]) (logid:d8c3ffe8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:39:45,404 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-335a1a93]) (logid:e054b4b5) HA health check task is running... 2025-06-29 02:39:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:39:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a337eda7]) (logid:c7cd292e) No inactive management server node found 2025-06-29 02:39:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a337eda7]) (logid:c7cd292e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b88e80c3]) (logid:636f220e) VM scheduler.poll is being called at 2025-06-29T00:40:00+0000 2025-06-29 02:39:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b88e80c3]) (logid:636f220e) Got 0 scheduled jobs to be executed at 2025-06-29T00:40:00+0000 2025-06-29 02:39:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b88e80c3]) (logid:636f220e) Cleaned up 0 VM scheduled job entries 2025-06-29 02:39:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-e749f359]) (logid:68f93155) VM Operation Thread Running 2025-06-29 02:39:46,329 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ecce5707]) (logid:c18a49ce) ClusterDRS.poll is being called at 2025-06-29T00:40:00+0000 2025-06-29 02:39:46,336 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ecce5707]) (logid:c18a49ce) Removed 0 old drs migration plans 2025-06-29 02:39:47,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:47,233 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:47,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:39:47,238 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:39:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c544c9a]) (logid:a53bbb7c) No inactive management server node found 2025-06-29 02:39:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c544c9a]) (logid:a53bbb7c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:48,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0524372]) (logid:a2adbcc8) No inactive management server node found 2025-06-29 02:39:48,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0524372]) (logid:a2adbcc8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:49,406 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2bf81580]) (logid:c3a7d96a) HA health check task is running... 2025-06-29 02:39:50,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7826: Seq 7-7826: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:50,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7826 2025-06-29 02:39:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe5f256a]) (logid:42ae48c0) No inactive management server node found 2025-06-29 02:39:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe5f256a]) (logid:42ae48c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b581192]) (logid:ad88c907) No inactive management server node found 2025-06-29 02:39:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b581192]) (logid:ad88c907) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-469fad9e]) (logid:ba51dc91) No inactive management server node found 2025-06-29 02:39:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-469fad9e]) (logid:ba51dc91) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:53,407 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4df38031]) (logid:f4728928) HA health check task is running... 2025-06-29 02:39:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0bdfd22]) (logid:49f9a737) No inactive management server node found 2025-06-29 02:39:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0bdfd22]) (logid:49f9a737) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:55,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7827: Seq 7-7827: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:39:55,045 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7827 2025-06-29 02:39:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8164fe8a]) (logid:0ab6786b) No inactive management server node found 2025-06-29 02:39:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8164fe8a]) (logid:0ab6786b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3cbb8d32]) (logid:fca01bdf) Enabled non-edge zones available for scan: 1 2025-06-29 02:39:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3cbb8d32]) (logid:fca01bdf) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:39:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3cbb8d32]) (logid:fca01bdf) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:39:56,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a7ff2478]) (logid:54eedfa8) Enabled non-edge zones available for scan: 1 2025-06-29 02:39:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a7ff2478]) (logid:54eedfa8) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:39:57,408 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-500a2c87]) (logid:4cbcab15) HA health check task is running... 2025-06-29 02:39:57,513 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:39:57,517 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:39:57,518 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) Seq 2-3048092522799696593: Routed from 90520738888109 2025-06-29 02:39:57,526 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 02:39:57,581 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) Seq 1-3048092522799696593: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:39:57,614 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:39:57,615 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) Seq 2-1612007191621929863: Routed from 90520738888109 2025-06-29 02:39:57,660 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4bd0c618]) (logid:5cb3deb2) Seq 2-1612007191621929863: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:39:57,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed15bfb3]) (logid:ecaa5664) No inactive management server node found 2025-06-29 02:39:57,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed15bfb3]) (logid:ecaa5664) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:39:57,856 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-d01349bf]) (logid:60a91bfb) AutoScaling Monitor is running... 2025-06-29 02:39:57,949 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-a012ae87]) (logid:b66243b7) Starting to remove all volume_stats rows older than [Sat Jun 28 14:39:57 CEST 2025]. 2025-06-29 02:39:57,949 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-a012ae87]) (logid:b66243b7) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:39:57 CEST 2025]. 2025-06-29 02:39:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a4f49ed]) (logid:5c0046ff) No inactive management server node found 2025-06-29 02:39:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a4f49ed]) (logid:5c0046ff) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d82eef6a]) (logid:2449fce9) No inactive management server node found 2025-06-29 02:40:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d82eef6a]) (logid:2449fce9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:01,410 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-21df712a]) (logid:83640162) HA health check task is running... 2025-06-29 02:40:02,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-280fc029]) (logid:4041f243) No inactive management server node found 2025-06-29 02:40:02,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-280fc029]) (logid:4041f243) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:03,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a528114]) (logid:7f026435) No inactive management server node found 2025-06-29 02:40:03,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a528114]) (logid:7f026435) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:05,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7828: Seq 7-7828: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:40:05,044 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7828 2025-06-29 02:40:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1384532]) (logid:ee52a2cc) No inactive management server node found 2025-06-29 02:40:05,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1384532]) (logid:ee52a2cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:05,411 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-66192335]) (logid:dc4b7bb1) HA health check task is running... 2025-06-29 02:40:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acfe52af]) (logid:a4f72a80) No inactive management server node found 2025-06-29 02:40:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acfe52af]) (logid:a4f72a80) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d424df8]) (logid:585ea66b) No inactive management server node found 2025-06-29 02:40:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d424df8]) (logid:585ea66b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:40:09,412 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-29ad7728]) (logid:e10bf525) HA health check task is running... 2025-06-29 02:40:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68ece9bb]) (logid:78177beb) No inactive management server node found 2025-06-29 02:40:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68ece9bb]) (logid:78177beb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5051c790]) (logid:355f763c) Found 0 routers to update status. 2025-06-29 02:40:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5051c790]) (logid:355f763c) Found 0 VPC's to update Redundant State. 2025-06-29 02:40:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5051c790]) (logid:355f763c) Found 0 networks to update RvR status. 2025-06-29 02:40:11,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb6c9203]) (logid:9e0b5671) No inactive management server node found 2025-06-29 02:40:11,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb6c9203]) (logid:9e0b5671) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:12,795 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-28d39b89]) (logid:70a04e32) Starting to remove all vm_stats rows older than [Sat Jun 28 14:40:12 CEST 2025]. 2025-06-29 02:40:12,810 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-28d39b89]) (logid:70a04e32) Removed a total of [0] vm_stats rows older than [Sat Jun 28 14:40:12 CEST 2025]. 2025-06-29 02:40:12,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c097a54]) (logid:f48c1775) No inactive management server node found 2025-06-29 02:40:12,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c097a54]) (logid:f48c1775) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:13,414 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6cbd30a5]) (logid:2c808d37) HA health check task is running... 2025-06-29 02:40:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2c64fa6]) (logid:af8ece8b) No inactive management server node found 2025-06-29 02:40:14,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2c64fa6]) (logid:af8ece8b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:15,037 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7829: Seq 7-7829: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:40:15,046 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7829 2025-06-29 02:40:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47fdf494]) (logid:ee43722b) No inactive management server node found 2025-06-29 02:40:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47fdf494]) (logid:ee43722b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:17,051 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d0621134]) (logid:64bd2b6c) Resetting hosts suitable for reconnect 2025-06-29 02:40:17,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d0621134]) (logid:64bd2b6c) Completed resetting hosts suitable for reconnect 2025-06-29 02:40:17,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d0621134]) (logid:64bd2b6c) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:40:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d0621134]) (logid:64bd2b6c) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:40:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d0621134]) (logid:64bd2b6c) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:40:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d0621134]) (logid:64bd2b6c) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:40:17,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73482caa]) (logid:88f2e50a) No inactive management server node found 2025-06-29 02:40:17,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73482caa]) (logid:88f2e50a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:40:17,415 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c7742a2f]) (logid:aad1db62) HA health check task is running... 2025-06-29 02:40:18,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e97ab927]) (logid:c218feaa) No inactive management server node found 2025-06-29 02:40:18,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e97ab927]) (logid:c218feaa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:20,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b122925c]) (logid:8a8be1e2) No inactive management server node found 2025-06-29 02:40:20,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b122925c]) (logid:8a8be1e2) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:40:21,416 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cab6b61d]) (logid:4b47521d) HA health check task is running... 2025-06-29 02:40:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c320f95c]) (logid:2fc01025) No inactive management server node found 2025-06-29 02:40:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c320f95c]) (logid:2fc01025) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:23,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9c525b9]) (logid:588b031f) No inactive management server node found 2025-06-29 02:40:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9c525b9]) (logid:588b031f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2c0b4b4]) (logid:cecd5c8e) No inactive management server node found 2025-06-29 02:40:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2c0b4b4]) (logid:cecd5c8e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:25,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7830: Seq 7-7830: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:40:25,045 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7830 2025-06-29 02:40:25,418 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8b452300]) (logid:60e6cdff) HA health check task is running... 2025-06-29 02:40:25,647 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) StorageCollector is running... 2025-06-29 02:40:25,653 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:40:25,653 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:40:25,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:40:25,655 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-6717118844223095457: Routed from 90520738888109 2025-06-29 02:40:25,709 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 4-6717118844223095457: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:40:25,712 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:40:25,712 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:40:25,712 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:40:25,713 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-1612007191621929864: Routed from 90520738888109 2025-06-29 02:40:25,724 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-1612007191621929864: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:40:25,745 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:40:25,745 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:40:25,745 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:40:25,746 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-1612007191621929865: Routed from 90520738888109 2025-06-29 02:40:25,837 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-1612007191621929865: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:40:25,847 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:40:25,847 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:40:25,847 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:40:25,848 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-3048092522799696594: Routed from 90520738888109 2025-06-29 02:40:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f4bd4ae]) (logid:84f3e02e) No inactive management server node found 2025-06-29 02:40:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f4bd4ae]) (logid:84f3e02e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6fe92f2a]) (logid:91a970bc) Enabled non-edge zones available for scan: 1 2025-06-29 02:40:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6fe92f2a]) (logid:91a970bc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:40:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6fe92f2a]) (logid:91a970bc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:40:26,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1901cd2f]) (logid:ed2e8073) Enabled non-edge zones available for scan: 1 2025-06-29 02:40:26,897 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1901cd2f]) (logid:ed2e8073) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:40:26,900 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 1-3048092522799696594: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:40:26,901 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:40:26,901 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:40:26,901 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:40:26,902 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-1612007191621929866: Routed from 90520738888109 2025-06-29 02:40:27,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-619c6123]) (logid:5f863dd2) No inactive management server node found 2025-06-29 02:40:27,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-619c6123]) (logid:5f863dd2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:27,961 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-97cdf1d7]) (logid:4189535e) Seq 2-1612007191621929866: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:40:29,143 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:40:29,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:40:29,145 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) Seq 2-3048092522799696595: Routed from 90520738888109 2025-06-29 02:40:29,148 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) Seq 1-3048092522799696595: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:40:29,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:40:29,149 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) Seq 2-1612007191621929867: Routed from 90520738888109 2025-06-29 02:40:29,152 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-06ea5e66]) (logid:c896750f) Seq 2-1612007191621929867: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:40:29,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa9d21e9]) (logid:34b76726) No inactive management server node found 2025-06-29 02:40:29,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa9d21e9]) (logid:34b76726) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:29,419 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b07f2f88]) (logid:90b88945) HA health check task is running... 2025-06-29 02:40:30,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a2ca4f2]) (logid:b5294bfb) No inactive management server node found 2025-06-29 02:40:30,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a2ca4f2]) (logid:b5294bfb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:32,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce06c5b3]) (logid:c9a73819) No inactive management server node found 2025-06-29 02:40:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce06c5b3]) (logid:c9a73819) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:33,420 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e44b12d4]) (logid:f8522fab) HA health check task is running... 2025-06-29 02:40:33,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edcc66de]) (logid:4e83f74b) No inactive management server node found 2025-06-29 02:40:33,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edcc66de]) (logid:4e83f74b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:33,984 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:40:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:40:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:40:33,989 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:40:35,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7831: Seq 7-7831: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:40:35,045 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7831 2025-06-29 02:40:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53bf8cd4]) (logid:77eb77aa) No inactive management server node found 2025-06-29 02:40:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53bf8cd4]) (logid:77eb77aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6da4574]) (logid:ffb95742) No inactive management server node found 2025-06-29 02:40:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6da4574]) (logid:ffb95742) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:37,421 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d07d58d8]) (logid:908825a5) HA health check task is running... 2025-06-29 02:40:38,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bedb344]) (logid:5d79fc64) No inactive management server node found 2025-06-29 02:40:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bedb344]) (logid:5d79fc64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e12ca6d]) (logid:9658d813) No inactive management server node found 2025-06-29 02:40:39,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e12ca6d]) (logid:9658d813) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:40,036 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7832: Seq 7-7832: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:40:40,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7832 2025-06-29 02:40:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ce18c443]) (logid:2c23efef) Found 0 routers to update status. 2025-06-29 02:40:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ce18c443]) (logid:2c23efef) Found 0 VPC's to update Redundant State. 2025-06-29 02:40:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ce18c443]) (logid:2c23efef) Found 0 networks to update RvR status. 2025-06-29 02:40:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-8a540649]) (logid:97fdf08b) DbCollector is running... 2025-06-29 02:40:41,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23ff3267]) (logid:4dd1ebe3) No inactive management server node found 2025-06-29 02:40:41,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23ff3267]) (logid:4dd1ebe3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:40:41,422 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ea13ffc7]) (logid:bd92a559) HA health check task is running... 2025-06-29 02:40:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) ManagementServerCollector is running... 2025-06-29 02:40:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) DB properties were already loaded 2025-06-29 02:40:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:40:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Execution of process [4061732] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:40:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Exit value of process [4061732] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:40:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Process [4061732] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:40:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) usage install: null 2025-06-29 02:40:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Sessions active: 0 2025-06-29 02:40:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:40:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061735] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:40:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:40:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061740] for command [/bin/bash -c uname -r ]. 2025-06-29 02:40:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:40:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061741] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:40:42,190 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) system memory from /proc: 4887515136 2025-06-29 02:40:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:40:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061746] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:40:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) free memory from /proc: 522268672 2025-06-29 02:40:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:40:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061751] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:40:42,209 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) used memory from /proc: 754828 2025-06-29 02:40:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:40:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061752] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:40:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:40:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061753] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:40:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:40:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061754] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:40:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:40:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061761] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:40:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:40:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Successfully executed process [4061762] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail buffersdirect.count, 120 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail buffersdirect.used, 657986 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail buffersmapped.count, 7 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail buffersmapped.used, 798602 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail gcPS-Scavenge.count, 473 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail gcPS-Scavenge.time, 3479 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail jvmuptime, 41582327 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memoryheap.committed, 348651520 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memoryheap.init, 77594624 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memoryheap.usage, 0.09986351912115275 2025-06-29 02:40:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorynon-heap.max, -1 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorynon-heap.usage, -2.2007288E8 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorynon-heap.used, 220072880 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17512829912023462 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.PS-Eden-Space.usage, 0.023101714303020993 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.PS-Old-Gen.usage, 0.11558773525465718 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorypools.PS-Survivor-Space.usage, 0.8318466913132441 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorytotal.committed, 573505536 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorytotal.init, 85262336 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorytotal.max, 1908932607 2025-06-29 02:40:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail memorytotal.used, 410762760 2025-06-29 02:40:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail threadsdeadlocks, [] 2025-06-29 02:40:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail threadsnew.count, 0 2025-06-29 02:40:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) not storing detail threadstimed_waiting.count, 84 2025-06-29 02:40:42,255 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-aa280514]) (logid:183c7ea3) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:40:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348651520,"freeJvmMemoryBytes":158787408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41582247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409936992,"totalCommitted":573505536,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190632728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522268672,"systemMemoryUsed":754828,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.31,0.24],"systemCyclesUsage":[66360138,53800935,4587366619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:40:42,255 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1385, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:40:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348651520,"freeJvmMemoryBytes":158787408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41582247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409936992,"totalCommitted":573505536,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190632728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522268672,"systemMemoryUsed":754828,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.31,0.24],"systemCyclesUsage":[66360138,53800935,4587366619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:40:42,255 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:40:42,260 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-310:[ctx-335f3fe6]) (logid:6fd7cb49) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:40:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348651520,"freeJvmMemoryBytes":158787408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41582247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409936992,"totalCommitted":573505536,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190632728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522268672,"systemMemoryUsed":754828,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.31,0.24],"systemCyclesUsage":[66360138,53800935,4587366619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:40:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:40:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1385, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:40:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":348651520,"freeJvmMemoryBytes":158787408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41582247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409936992,"totalCommitted":573505536,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190632728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522268672,"systemMemoryUsed":754828,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.31,0.24],"systemCyclesUsage":[66360138,53800935,4587366619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:40:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67131709]) (logid:4ef8d255) No inactive management server node found 2025-06-29 02:40:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67131709]) (logid:4ef8d255) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24aea3b8]) (logid:d9a62e5b) No inactive management server node found 2025-06-29 02:40:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24aea3b8]) (logid:d9a62e5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:45,339 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:40:45,424 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-66d9015b]) (logid:f2997e69) HA health check task is running... 2025-06-29 02:40:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:40:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c7447b3]) (logid:b823a0a6) No inactive management server node found 2025-06-29 02:40:45,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c7447b3]) (logid:b823a0a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c946d221]) (logid:f920b324) VM scheduler.poll is being called at 2025-06-29T00:41:00+0000 2025-06-29 02:40:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c946d221]) (logid:f920b324) Got 0 scheduled jobs to be executed at 2025-06-29T00:41:00+0000 2025-06-29 02:40:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c946d221]) (logid:f920b324) Cleaned up 0 VM scheduled job entries 2025-06-29 02:40:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-5b1006aa]) (logid:f2f44b21) VM Operation Thread Running 2025-06-29 02:40:46,329 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7623119f]) (logid:a694fb1d) ClusterDRS.poll is being called at 2025-06-29T00:41:00+0000 2025-06-29 02:40:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7623119f]) (logid:a694fb1d) Removed 0 old drs migration plans 2025-06-29 02:40:47,241 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:40:47,243 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:40:47,244 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:40:47,247 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:40:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-833f3c85]) (logid:b4180948) No inactive management server node found 2025-06-29 02:40:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-833f3c85]) (logid:b4180948) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:48,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f717c2b3]) (logid:a65cd897) No inactive management server node found 2025-06-29 02:40:48,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f717c2b3]) (logid:a65cd897) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:49,425 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-aa38e9f0]) (logid:010265ff) HA health check task is running... 2025-06-29 02:40:50,036 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7834: Seq 7-7834: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:40:50,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7834 2025-06-29 02:40:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d164899]) (logid:c3311f2b) No inactive management server node found 2025-06-29 02:40:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d164899]) (logid:c3311f2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d15ff84c]) (logid:40241b37) No inactive management server node found 2025-06-29 02:40:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d15ff84c]) (logid:40241b37) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:53,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e2e053b]) (logid:21f9e493) No inactive management server node found 2025-06-29 02:40:53,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e2e053b]) (logid:21f9e493) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:40:53,426 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8ffc1a39]) (logid:7b41ba45) HA health check task is running... 2025-06-29 02:40:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a9fa87]) (logid:aff7ae4d) No inactive management server node found 2025-06-29 02:40:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a9fa87]) (logid:aff7ae4d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-917c75ec]) (logid:1dd9a713) No inactive management server node found 2025-06-29 02:40:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-917c75ec]) (logid:1dd9a713) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fc2a490f]) (logid:4069bd3d) Enabled non-edge zones available for scan: 1 2025-06-29 02:40:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-fc2a490f]) (logid:4069bd3d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:40:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fc2a490f]) (logid:4069bd3d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:40:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-af29ecf4]) (logid:438b581e) Enabled non-edge zones available for scan: 1 2025-06-29 02:40:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-af29ecf4]) (logid:438b581e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:40:57,427 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-54bdba6e]) (logid:0afa79b2) HA health check task is running... 2025-06-29 02:40:57,714 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:40:57,718 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:40:57,719 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) Seq 2-3048092522799696596: Routed from 90520738888109 2025-06-29 02:40:57,773 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) Seq 1-3048092522799696596: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:40:57,824 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:40:57,825 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) Seq 2-1612007191621929868: Routed from 90520738888109 2025-06-29 02:40:57,825 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-139f8271]) (logid:750cce6c) No inactive management server node found 2025-06-29 02:40:57,825 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-139f8271]) (logid:750cce6c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:40:57,858 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-911a61f6]) (logid:1e51a602) AutoScaling Monitor is running... 2025-06-29 02:40:57,873 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-513a597d]) (logid:1625ebe6) Seq 2-1612007191621929868: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:40:57,951 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-75f9d4e1]) (logid:ad9c5195) Starting to remove all volume_stats rows older than [Sat Jun 28 14:40:57 CEST 2025]. 2025-06-29 02:40:57,952 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-75f9d4e1]) (logid:ad9c5195) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:40:57 CEST 2025]. 2025-06-29 02:40:59,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ab149e9]) (logid:b5a02d98) No inactive management server node found 2025-06-29 02:40:59,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ab149e9]) (logid:b5a02d98) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:00,036 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7835: Seq 7-7835: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:00,058 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7835 2025-06-29 02:41:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0bbd074]) (logid:e0a3c0c2) No inactive management server node found 2025-06-29 02:41:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0bbd074]) (logid:e0a3c0c2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:01,428 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8364e141]) (logid:7abfdf66) HA health check task is running... 2025-06-29 02:41:02,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edc2c964]) (logid:7ded756e) No inactive management server node found 2025-06-29 02:41:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edc2c964]) (logid:7ded756e) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:41:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eaa68a7]) (logid:34e312af) No inactive management server node found 2025-06-29 02:41:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eaa68a7]) (logid:34e312af) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cabad00]) (logid:85900bfa) No inactive management server node found 2025-06-29 02:41:05,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cabad00]) (logid:85900bfa) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:05,429 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-bd24369e]) (logid:2c9b78f8) HA health check task is running... 2025-06-29 02:41:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ce71849]) (logid:117ae4c6) No inactive management server node found 2025-06-29 02:41:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ce71849]) (logid:117ae4c6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a1cdc89]) (logid:d418e355) No inactive management server node found 2025-06-29 02:41:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a1cdc89]) (logid:d418e355) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:09,431 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5f3c7acf]) (logid:8971375a) HA health check task is running... 2025-06-29 02:41:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9750691]) (logid:08484bb1) No inactive management server node found 2025-06-29 02:41:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9750691]) (logid:08484bb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:10,037 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7836: Seq 7-7836: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:10,046 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7836 2025-06-29 02:41:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d6300840]) (logid:2aabe864) Found 0 routers to update status. 2025-06-29 02:41:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d6300840]) (logid:2aabe864) Found 0 VPC's to update Redundant State. 2025-06-29 02:41:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d6300840]) (logid:2aabe864) Found 0 networks to update RvR status. 2025-06-29 02:41:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bab3a8c]) (logid:5d894417) No inactive management server node found 2025-06-29 02:41:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bab3a8c]) (logid:5d894417) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:12,811 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-db99d8bf]) (logid:e001f815) Starting to remove all vm_stats rows older than [Sat Jun 28 14:41:12 CEST 2025]. 2025-06-29 02:41:12,828 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-db99d8bf]) (logid:e001f815) Removed a total of [0] vm_stats rows older than [Sat Jun 28 14:41:12 CEST 2025]. 2025-06-29 02:41:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9d73fb3]) (logid:4b474484) No inactive management server node found 2025-06-29 02:41:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9d73fb3]) (logid:4b474484) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:13,432 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-32b9150d]) (logid:3199330e) HA health check task is running... 2025-06-29 02:41:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27bb5b38]) (logid:1b09dfee) No inactive management server node found 2025-06-29 02:41:14,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27bb5b38]) (logid:1b09dfee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:15,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78e9a13e]) (logid:efdf2ab2) No inactive management server node found 2025-06-29 02:41:15,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78e9a13e]) (logid:efdf2ab2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:17,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36c75509]) (logid:72c945cf) No inactive management server node found 2025-06-29 02:41:17,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36c75509]) (logid:72c945cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:17,433 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-03c9e9d9]) (logid:20372fa7) HA health check task is running... 2025-06-29 02:41:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b8e7dae]) (logid:b9761db8) No inactive management server node found 2025-06-29 02:41:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b8e7dae]) (logid:b9761db8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:20,037 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7837: Seq 7-7837: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:20,047 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7837 2025-06-29 02:41:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16660108]) (logid:56d0226f) No inactive management server node found 2025-06-29 02:41:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16660108]) (logid:56d0226f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:21,435 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-24fb7f88]) (logid:339ed37a) HA health check task is running... 2025-06-29 02:41:21,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eed27d3e]) (logid:055df38a) No inactive management server node found 2025-06-29 02:41:21,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eed27d3e]) (logid:055df38a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:23,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0ca6be8]) (logid:5caed6ad) No inactive management server node found 2025-06-29 02:41:23,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0ca6be8]) (logid:5caed6ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb44fb3d]) (logid:13479f45) No inactive management server node found 2025-06-29 02:41:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb44fb3d]) (logid:13479f45) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:25,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7838: Seq 7-7838: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:25,048 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7838 2025-06-29 02:41:25,436 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-258b020b]) (logid:4a34068d) HA health check task is running... 2025-06-29 02:41:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96b5b3b4]) (logid:15fbf38e) No inactive management server node found 2025-06-29 02:41:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96b5b3b4]) (logid:15fbf38e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c108a78]) (logid:e68e4c0a) Enabled non-edge zones available for scan: 1 2025-06-29 02:41:26,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0c108a78]) (logid:e68e4c0a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:41:26,765 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c108a78]) (logid:e68e4c0a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:41:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-de703d23]) (logid:f7a2620b) Enabled non-edge zones available for scan: 1 2025-06-29 02:41:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-de703d23]) (logid:f7a2620b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:41:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0cc7358]) (logid:aa85b6f0) No inactive management server node found 2025-06-29 02:41:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0cc7358]) (logid:aa85b6f0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:27,961 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) StorageCollector is running... 2025-06-29 02:41:27,966 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:41:27,966 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:41:27,966 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:41:27,968 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 2-6717118844223095458: Routed from 90520738888109 2025-06-29 02:41:28,022 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 4-6717118844223095458: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:41:28,026 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:41:28,026 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:41:28,026 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:41:28,027 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 2-3048092522799696597: Routed from 90520738888109 2025-06-29 02:41:28,075 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 1-3048092522799696597: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:41:28,085 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:41:28,085 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:41:28,085 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:41:28,086 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 2-3048092522799696598: Routed from 90520738888109 2025-06-29 02:41:28,173 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 1-3048092522799696598: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:41:28,183 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:41:28,184 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:41:28,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:41:28,184 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 2-3048092522799696599: Routed from 90520738888109 2025-06-29 02:41:29,155 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:41:29,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:41:29,157 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) Seq 2-3048092522799696600: Routed from 90520738888109 2025-06-29 02:41:29,200 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) Seq 1-3048092522799696600: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:41:29,202 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:41:29,203 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) Seq 2-1612007191621929869: Routed from 90520738888109 2025-06-29 02:41:29,205 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9540fb9b]) (logid:aec73129) Seq 2-1612007191621929869: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:41:29,235 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 1-3048092522799696599: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:41:29,237 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:41:29,237 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:41:29,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:41:29,238 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 2-1612007191621929870: Routed from 90520738888109 2025-06-29 02:41:29,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f169c6ef]) (logid:d7188045) No inactive management server node found 2025-06-29 02:41:29,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f169c6ef]) (logid:d7188045) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:29,437 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c4f7f89e]) (logid:6fc6f176) HA health check task is running... 2025-06-29 02:41:30,345 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ad810551]) (logid:78af04c3) Seq 2-1612007191621929870: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:41:30,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-caa5decd]) (logid:13893e71) No inactive management server node found 2025-06-29 02:41:30,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-caa5decd]) (logid:13893e71) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b947a86]) (logid:e1e9bf16) No inactive management server node found 2025-06-29 02:41:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b947a86]) (logid:e1e9bf16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:33,438 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6ba6de91]) (logid:5afa92db) HA health check task is running... 2025-06-29 02:41:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f802e13]) (logid:8e3026df) No inactive management server node found 2025-06-29 02:41:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f802e13]) (logid:8e3026df) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:41:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:41:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:41:33,989 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:41:35,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7839: Seq 7-7839: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:35,059 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7839 2025-06-29 02:41:35,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9a31aa8]) (logid:05906750) No inactive management server node found 2025-06-29 02:41:35,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9a31aa8]) (logid:05906750) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:41:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cceac61d]) (logid:1464c466) No inactive management server node found 2025-06-29 02:41:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cceac61d]) (logid:1464c466) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:37,440 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c42bef90]) (logid:b96e2f2a) HA health check task is running... 2025-06-29 02:41:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f4c52de]) (logid:0c194258) No inactive management server node found 2025-06-29 02:41:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f4c52de]) (logid:0c194258) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1130a2d]) (logid:fde01c6d) No inactive management server node found 2025-06-29 02:41:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1130a2d]) (logid:fde01c6d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-aa8eff29]) (logid:e3ccc5d0) Found 0 routers to update status. 2025-06-29 02:41:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-aa8eff29]) (logid:e3ccc5d0) Found 0 VPC's to update Redundant State. 2025-06-29 02:41:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-aa8eff29]) (logid:e3ccc5d0) Found 0 networks to update RvR status. 2025-06-29 02:41:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-c88c6c2b]) (logid:5a7b1455) DbCollector is running... 2025-06-29 02:41:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1c80b27]) (logid:e1f2d4ab) No inactive management server node found 2025-06-29 02:41:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1c80b27]) (logid:e1f2d4ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:41,441 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-edb67432]) (logid:64ec4789) HA health check task is running... 2025-06-29 02:41:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) ManagementServerCollector is running... 2025-06-29 02:41:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) DB properties were already loaded 2025-06-29 02:41:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:41:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Execution of process [4061768] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:41:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Exit value of process [4061768] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:41:42,169 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Process [4061768] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:41:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) usage install: null 2025-06-29 02:41:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Sessions active: 0 2025-06-29 02:41:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:41:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061771] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:41:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:41:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061776] for command [/bin/bash -c uname -r ]. 2025-06-29 02:41:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:41:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061777] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:41:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) system memory from /proc: 4887515136 2025-06-29 02:41:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:41:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061782] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:41:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) free memory from /proc: 524079104 2025-06-29 02:41:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:41:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061787] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:41:42,213 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) used memory from /proc: 754956 2025-06-29 02:41:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:41:42,219 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061788] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:41:42,219 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:41:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061789] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:41:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:41:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061790] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:41:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:41:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061797] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:41:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:41:42,258 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Successfully executed process [4061798] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:41:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail buffersdirect.count, 120 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail buffersdirect.used, 657986 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail buffersmapped.count, 7 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail buffersmapped.used, 798602 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail gcPS-Scavenge.count, 474 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail gcPS-Scavenge.time, 3486 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail jvmuptime, 41642337 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memoryheap.committed, 349175808 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memoryheap.init, 77594624 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memoryheap.usage, 0.09902566031288623 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorynon-heap.max, -1 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorynon-heap.usage, -2.20101296E8 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorynon-heap.used, 220101296 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17535948747000799 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.PS-Eden-Space.usage, 0.02094226027979995 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.PS-Old-Gen.usage, 0.11558773525465718 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorypools.PS-Survivor-Space.usage, 0.7869366732510653 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorytotal.committed, 574029824 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorytotal.init, 85262336 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorytotal.max, 1908932607 2025-06-29 02:41:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail memorytotal.used, 409192840 2025-06-29 02:41:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail threadsdeadlocks, [] 2025-06-29 02:41:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail threadsnew.count, 0 2025-06-29 02:41:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) not storing detail threadstimed_waiting.count, 84 2025-06-29 02:41:42,264 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-c75c7ada]) (logid:a164ea54) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:41:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349175808,"freeJvmMemoryBytes":160899056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41642246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":408397456,"totalCommitted":574029824,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189033312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":524079104,"systemMemoryUsed":754956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.3,0.24],"systemCyclesUsage":[66360477,53801255,4587389710],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:41:42,264 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1387, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:41:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349175808,"freeJvmMemoryBytes":160899056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41642246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":408397456,"totalCommitted":574029824,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189033312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":524079104,"systemMemoryUsed":754956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.3,0.24],"systemCyclesUsage":[66360477,53801255,4587389710],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:41:42,264 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:41:42,268 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:41:42,268 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1387, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:41:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349175808,"freeJvmMemoryBytes":160899056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41642246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":408397456,"totalCommitted":574029824,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189033312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":524079104,"systemMemoryUsed":754956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.3,0.24],"systemCyclesUsage":[66360477,53801255,4587389710],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:41:42,268 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-311:[ctx-6adb472a]) (logid:8ab384b3) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:41:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349175808,"freeJvmMemoryBytes":160899056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41642246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":408397456,"totalCommitted":574029824,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189033312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":524079104,"systemMemoryUsed":754956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.3,0.24],"systemCyclesUsage":[66360477,53801255,4587389710],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:41:42,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5708a5fb]) (logid:c51433c5) No inactive management server node found 2025-06-29 02:41:42,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5708a5fb]) (logid:c51433c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:44,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-207156f6]) (logid:69dcdbd2) No inactive management server node found 2025-06-29 02:41:44,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-207156f6]) (logid:69dcdbd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:45,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7840: Seq 7-7840: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:45,048 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7840 2025-06-29 02:41:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:41:45,442 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-8ef08962]) (logid:5aa568ad) HA health check task is running... 2025-06-29 02:41:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:41:45,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48a31e33]) (logid:17195ba6) No inactive management server node found 2025-06-29 02:41:45,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48a31e33]) (logid:17195ba6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ef07e04d]) (logid:4ade1648) VM scheduler.poll is being called at 2025-06-29T00:42:00+0000 2025-06-29 02:41:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ef07e04d]) (logid:4ade1648) Got 0 scheduled jobs to be executed at 2025-06-29T00:42:00+0000 2025-06-29 02:41:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ef07e04d]) (logid:4ade1648) Cleaned up 0 VM scheduled job entries 2025-06-29 02:41:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-dd7302de]) (logid:b265b73e) VM Operation Thread Running 2025-06-29 02:41:46,329 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1d383a56]) (logid:4a67f13a) ClusterDRS.poll is being called at 2025-06-29T00:42:00+0000 2025-06-29 02:41:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1d383a56]) (logid:4a67f13a) Removed 0 old drs migration plans 2025-06-29 02:41:47,051 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-58eafc12]) (logid:dbb31810) Resetting hosts suitable for reconnect 2025-06-29 02:41:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-58eafc12]) (logid:dbb31810) Completed resetting hosts suitable for reconnect 2025-06-29 02:41:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-58eafc12]) (logid:dbb31810) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:41:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-58eafc12]) (logid:dbb31810) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:41:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-58eafc12]) (logid:dbb31810) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:41:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-58eafc12]) (logid:dbb31810) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:41:47,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:41:47,236 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:41:47,238 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:41:47,240 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:41:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de2fc77e]) (logid:e29ccab5) No inactive management server node found 2025-06-29 02:41:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de2fc77e]) (logid:e29ccab5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d06efc26]) (logid:aacf21a6) No inactive management server node found 2025-06-29 02:41:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d06efc26]) (logid:aacf21a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:49,443 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3040d4ef]) (logid:3454745c) HA health check task is running... 2025-06-29 02:41:50,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7842: Seq 7-7842: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:41:50,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7842 2025-06-29 02:41:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-935ef334]) (logid:2f00b519) No inactive management server node found 2025-06-29 02:41:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-935ef334]) (logid:2f00b519) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a0ce82e]) (logid:696e26ae) No inactive management server node found 2025-06-29 02:41:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a0ce82e]) (logid:696e26ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:53,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6e35d68]) (logid:e7ae8332) No inactive management server node found 2025-06-29 02:41:53,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6e35d68]) (logid:e7ae8332) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:53,444 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d4131545]) (logid:f272f97f) HA health check task is running... 2025-06-29 02:41:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b801c368]) (logid:cff0aa6b) No inactive management server node found 2025-06-29 02:41:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b801c368]) (logid:cff0aa6b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef9689b4]) (logid:d9b06119) No inactive management server node found 2025-06-29 02:41:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef9689b4]) (logid:d9b06119) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-612699f1]) (logid:a68cd815) Enabled non-edge zones available for scan: 1 2025-06-29 02:41:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-612699f1]) (logid:a68cd815) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:41:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-612699f1]) (logid:a68cd815) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:41:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3081a4e5]) (logid:0bfc83c0) Enabled non-edge zones available for scan: 1 2025-06-29 02:41:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3081a4e5]) (logid:0bfc83c0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:41:57,446 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6d499d93]) (logid:875ac579) HA health check task is running... 2025-06-29 02:41:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6598bd8]) (logid:98405728) No inactive management server node found 2025-06-29 02:41:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6598bd8]) (logid:98405728) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:41:57,859 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-ad75d08a]) (logid:cba44f91) AutoScaling Monitor is running... 2025-06-29 02:41:57,934 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:41:57,937 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:41:57,938 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) Seq 2-3048092522799696601: Routed from 90520738888109 2025-06-29 02:41:57,954 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b9dbd3ca]) (logid:d269cbf1) Starting to remove all volume_stats rows older than [Sat Jun 28 14:41:57 CEST 2025]. 2025-06-29 02:41:57,954 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b9dbd3ca]) (logid:d269cbf1) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:41:57 CEST 2025]. 2025-06-29 02:41:57,990 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) Seq 1-3048092522799696601: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:41:58,027 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:41:58,028 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) Seq 2-1612007191621929871: Routed from 90520738888109 2025-06-29 02:41:58,084 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b4ddc9fe]) (logid:d8de80eb) Seq 2-1612007191621929871: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:41:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6c1dda0]) (logid:56afa93a) No inactive management server node found 2025-06-29 02:41:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6c1dda0]) (logid:56afa93a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:00,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7843: Seq 7-7843: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:00,060 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7843 2025-06-29 02:42:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3546fef4]) (logid:3fad7828) No inactive management server node found 2025-06-29 02:42:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3546fef4]) (logid:3fad7828) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:01,447 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-63651803]) (logid:c9c99b2d) HA health check task is running... 2025-06-29 02:42:02,345 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff182402]) (logid:0050c337) No inactive management server node found 2025-06-29 02:42:02,345 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff182402]) (logid:0050c337) Peer scan is finished. profiler: Done. Duration: 11ms , profilerQueryActiveList: Done. Duration: 10ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cbfc930]) (logid:af30a9b9) No inactive management server node found 2025-06-29 02:42:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cbfc930]) (logid:af30a9b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc9df69e]) (logid:576f49d7) No inactive management server node found 2025-06-29 02:42:05,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc9df69e]) (logid:576f49d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:05,448 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a9e6c160]) (logid:0f450031) HA health check task is running... 2025-06-29 02:42:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b6718a5]) (logid:a642a968) No inactive management server node found 2025-06-29 02:42:06,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b6718a5]) (logid:a642a968) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cc4fe24]) (logid:22e41b03) No inactive management server node found 2025-06-29 02:42:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cc4fe24]) (logid:22e41b03) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:09,450 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-79fe11ad]) (logid:070b5037) HA health check task is running... 2025-06-29 02:42:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de6a28d5]) (logid:12e5cdaa) No inactive management server node found 2025-06-29 02:42:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de6a28d5]) (logid:12e5cdaa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:10,041 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7844: Seq 7-7844: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:10,050 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7844 2025-06-29 02:42:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-84b4d30b]) (logid:c22642fb) Found 0 routers to update status. 2025-06-29 02:42:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-84b4d30b]) (logid:c22642fb) Found 0 VPC's to update Redundant State. 2025-06-29 02:42:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-84b4d30b]) (logid:c22642fb) Found 0 networks to update RvR status. 2025-06-29 02:42:11,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e55f4af]) (logid:c7ac745a) No inactive management server node found 2025-06-29 02:42:11,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e55f4af]) (logid:c7ac745a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:42:12,830 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-3eec0d28]) (logid:618cde51) Starting to remove all vm_stats rows older than [Sat Jun 28 14:42:12 CEST 2025]. 2025-06-29 02:42:12,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a446f92d]) (logid:9dd7b179) No inactive management server node found 2025-06-29 02:42:12,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a446f92d]) (logid:9dd7b179) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:12,848 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-3eec0d28]) (logid:618cde51) Removed a total of [0] vm_stats rows older than [Sat Jun 28 14:42:12 CEST 2025]. 2025-06-29 02:42:13,451 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c3b5e68f]) (logid:ad60dd0c) HA health check task is running... 2025-06-29 02:42:14,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b08ff48]) (logid:d9bbf059) No inactive management server node found 2025-06-29 02:42:14,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b08ff48]) (logid:d9bbf059) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f9e1410]) (logid:5e1df0aa) No inactive management server node found 2025-06-29 02:42:15,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f9e1410]) (logid:5e1df0aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:17,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d0d2f0c]) (logid:7f6cbe6f) No inactive management server node found 2025-06-29 02:42:17,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d0d2f0c]) (logid:7f6cbe6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:17,452 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-42ac1519]) (logid:a12bbc84) HA health check task is running... 2025-06-29 02:42:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-436f6746]) (logid:954c190f) No inactive management server node found 2025-06-29 02:42:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-436f6746]) (logid:954c190f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:20,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7845: Seq 7-7845: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:20,050 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7845 2025-06-29 02:42:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfa4010f]) (logid:7d0fcc18) No inactive management server node found 2025-06-29 02:42:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfa4010f]) (logid:7d0fcc18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:21,454 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-44d8ec13]) (logid:a8e36b8c) HA health check task is running... 2025-06-29 02:42:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58db86fc]) (logid:7a992b4d) No inactive management server node found 2025-06-29 02:42:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58db86fc]) (logid:7a992b4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7991c9c]) (logid:a1e5d2b8) No inactive management server node found 2025-06-29 02:42:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7991c9c]) (logid:a1e5d2b8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6c58eee]) (logid:a6f12178) No inactive management server node found 2025-06-29 02:42:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6c58eee]) (logid:a6f12178) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:25,455 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-41a48ab9]) (logid:7da1ba22) HA health check task is running... 2025-06-29 02:42:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bd89c0c]) (logid:ac757db1) No inactive management server node found 2025-06-29 02:42:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bd89c0c]) (logid:ac757db1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-16d831f9]) (logid:62fb7cfa) Enabled non-edge zones available for scan: 1 2025-06-29 02:42:26,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-16d831f9]) (logid:62fb7cfa) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:42:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-16d831f9]) (logid:62fb7cfa) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:42:26,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-33219e15]) (logid:cf71b9ba) Enabled non-edge zones available for scan: 1 2025-06-29 02:42:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-33219e15]) (logid:cf71b9ba) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:42:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3911562]) (logid:3bd80fca) No inactive management server node found 2025-06-29 02:42:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3911562]) (logid:3bd80fca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:29,208 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:42:29,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:42:29,211 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) Seq 2-3048092522799696602: Routed from 90520738888109 2025-06-29 02:42:29,254 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) Seq 1-3048092522799696602: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:42:29,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:42:29,256 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) Seq 2-1612007191621929872: Routed from 90520738888109 2025-06-29 02:42:29,259 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ff13c09b]) (logid:a5233c83) Seq 2-1612007191621929872: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:42:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9b59e4a]) (logid:99c539fb) No inactive management server node found 2025-06-29 02:42:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9b59e4a]) (logid:99c539fb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:29,456 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-41eb674d]) (logid:184acd24) HA health check task is running... 2025-06-29 02:42:30,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7846: Seq 7-7846: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:30,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7846 2025-06-29 02:42:30,346 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) StorageCollector is running... 2025-06-29 02:42:30,351 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:42:30,351 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:42:30,351 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:42:30,352 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-6717118844223095459: Routed from 90520738888109 2025-06-29 02:42:30,415 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 4-6717118844223095459: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:42:30,418 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:42:30,418 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:42:30,418 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:42:30,419 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-3048092522799696603: Routed from 90520738888109 2025-06-29 02:42:30,468 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 1-3048092522799696603: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:42:30,492 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:42:30,492 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:42:30,492 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:42:30,493 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-1612007191621929873: Routed from 90520738888109 2025-06-29 02:42:30,551 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-1612007191621929873: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:42:30,563 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:42:30,563 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:42:30,563 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:42:30,564 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-3048092522799696604: Routed from 90520738888109 2025-06-29 02:42:30,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-020a5d51]) (logid:af9ea89e) No inactive management server node found 2025-06-29 02:42:30,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-020a5d51]) (logid:af9ea89e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:31,615 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 1-3048092522799696604: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:42:31,617 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:42:31,617 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:42:31,617 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:42:31,618 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-1612007191621929874: Routed from 90520738888109 2025-06-29 02:42:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a494037b]) (logid:0da76c74) No inactive management server node found 2025-06-29 02:42:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a494037b]) (logid:0da76c74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:32,682 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a365c624]) (logid:ffff491b) Seq 2-1612007191621929874: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:42:33,457 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a318030a]) (logid:bb4ef28c) HA health check task is running... 2025-06-29 02:42:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84598ac7]) (logid:e8ead352) No inactive management server node found 2025-06-29 02:42:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84598ac7]) (logid:e8ead352) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:33,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:42:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:42:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:42:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:42:35,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7847: Seq 7-7847: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:35,051 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7847 2025-06-29 02:42:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-625053ff]) (logid:29ce62f5) No inactive management server node found 2025-06-29 02:42:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-625053ff]) (logid:29ce62f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a353187]) (logid:5963f2c8) No inactive management server node found 2025-06-29 02:42:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a353187]) (logid:5963f2c8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:37,459 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-696467f3]) (logid:a952f41c) HA health check task is running... 2025-06-29 02:42:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-740fb785]) (logid:c74ec200) No inactive management server node found 2025-06-29 02:42:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-740fb785]) (logid:c74ec200) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cf6388c]) (logid:ece6c873) No inactive management server node found 2025-06-29 02:42:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cf6388c]) (logid:ece6c873) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d48acf4]) (logid:3241d4f3) Found 0 routers to update status. 2025-06-29 02:42:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d48acf4]) (logid:3241d4f3) Found 0 VPC's to update Redundant State. 2025-06-29 02:42:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d48acf4]) (logid:3241d4f3) Found 0 networks to update RvR status. 2025-06-29 02:42:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-7eff8739]) (logid:cebdc99c) DbCollector is running... 2025-06-29 02:42:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dfa12e2]) (logid:0f80a9e0) No inactive management server node found 2025-06-29 02:42:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dfa12e2]) (logid:0f80a9e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:41,460 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cc35d329]) (logid:257d8c48) HA health check task is running... 2025-06-29 02:42:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) ManagementServerCollector is running... 2025-06-29 02:42:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) DB properties were already loaded 2025-06-29 02:42:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:42:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Execution of process [4061812] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:42:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Exit value of process [4061812] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:42:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Process [4061812] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:42:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) usage install: null 2025-06-29 02:42:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Sessions active: 0 2025-06-29 02:42:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:42:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061815] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:42:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:42:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061820] for command [/bin/bash -c uname -r ]. 2025-06-29 02:42:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:42:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061821] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:42:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) system memory from /proc: 4887515136 2025-06-29 02:42:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:42:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061826] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:42:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) free memory from /proc: 522014720 2025-06-29 02:42:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:42:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061831] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:42:42,208 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) used memory from /proc: 755852 2025-06-29 02:42:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:42:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061832] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:42:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:42:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061833] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:42:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:42:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061834] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:42:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:42:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061841] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:42:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:42:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Successfully executed process [4061842] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:42:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:42:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail buffersdirect.count, 120 2025-06-29 02:42:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail buffersdirect.used, 657986 2025-06-29 02:42:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:42:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail buffersmapped.count, 7 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail buffersmapped.used, 798602 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail gcPS-Scavenge.count, 475 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail gcPS-Scavenge.time, 3494 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail jvmuptime, 41702327 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memoryheap.committed, 349700096 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memoryheap.init, 77594624 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memoryheap.usage, 0.09845078826376254 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorynon-heap.max, -1 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorynon-heap.usage, -2.20101296E8 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorynon-heap.used, 220101296 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17535948747000799 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.PS-Eden-Space.usage, 0.0193676753181608 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.PS-Old-Gen.usage, 0.11558773525465718 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorypools.PS-Survivor-Space.usage, 0.7527220352836277 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorytotal.committed, 574554112 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorytotal.init, 85262336 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorytotal.max, 1908932607 2025-06-29 02:42:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail memorytotal.used, 408089648 2025-06-29 02:42:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail threadsdeadlocks, [] 2025-06-29 02:42:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail threadsnew.count, 0 2025-06-29 02:42:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:42:42,254 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-5b6d9cdd]) (logid:5c28d1c9) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:42:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349700096,"freeJvmMemoryBytes":162523968,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41702247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.3,"totalInit":85262336,"totalUsed":407277424,"totalCommitted":574554112,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187935920,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522014720,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.3,0.31,0.25],"systemCyclesUsage":[66360816,53801594,4587412760],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:42:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1389, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:42:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349700096,"freeJvmMemoryBytes":162523968,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41702247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.3,"totalInit":85262336,"totalUsed":407277424,"totalCommitted":574554112,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187935920,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522014720,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.3,0.31,0.25],"systemCyclesUsage":[66360816,53801594,4587412760],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:42:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:42:42,258 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-311:[ctx-26e8b463]) (logid:874d2e2c) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:42:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349700096,"freeJvmMemoryBytes":162523968,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41702247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.3,"totalInit":85262336,"totalUsed":407277424,"totalCommitted":574554112,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187935920,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522014720,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.3,0.31,0.25],"systemCyclesUsage":[66360816,53801594,4587412760],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:42:42,258 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:42:42,258 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1389, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:42:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":349700096,"freeJvmMemoryBytes":162523968,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41702247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.3,"totalInit":85262336,"totalUsed":407277424,"totalCommitted":574554112,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187935920,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522014720,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.3,0.31,0.25],"systemCyclesUsage":[66360816,53801594,4587412760],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:42:42,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-311fcfea]) (logid:3088792f) No inactive management server node found 2025-06-29 02:42:42,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-311fcfea]) (logid:3088792f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c144f83]) (logid:fa61555c) No inactive management server node found 2025-06-29 02:42:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c144f83]) (logid:fa61555c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:45,041 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7848: Seq 7-7848: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:45,050 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7848 2025-06-29 02:42:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:42:45,461 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-509fc603]) (logid:0f8d4db3) HA health check task is running... 2025-06-29 02:42:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:42:45,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c821bd82]) (logid:555b7d8c) No inactive management server node found 2025-06-29 02:42:45,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c821bd82]) (logid:555b7d8c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ddf97c40]) (logid:5d4b6da5) VM scheduler.poll is being called at 2025-06-29T00:43:00+0000 2025-06-29 02:42:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ddf97c40]) (logid:5d4b6da5) Got 0 scheduled jobs to be executed at 2025-06-29T00:43:00+0000 2025-06-29 02:42:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ddf97c40]) (logid:5d4b6da5) Cleaned up 0 VM scheduled job entries 2025-06-29 02:42:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-01cfa8ba]) (logid:74d92ffe) VM Operation Thread Running 2025-06-29 02:42:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-bdceb8a5]) (logid:6b54ffb2) ClusterDRS.poll is being called at 2025-06-29T00:43:00+0000 2025-06-29 02:42:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-bdceb8a5]) (logid:6b54ffb2) Removed 0 old drs migration plans 2025-06-29 02:42:47,221 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:42:47,222 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:42:47,224 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:42:47,227 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:42:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb1938bd]) (logid:6805d341) No inactive management server node found 2025-06-29 02:42:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb1938bd]) (logid:6805d341) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba77ec81]) (logid:6313d9c7) No inactive management server node found 2025-06-29 02:42:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba77ec81]) (logid:6313d9c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:49,462 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6ce71be7]) (logid:78437ec8) HA health check task is running... 2025-06-29 02:42:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b639ed56]) (logid:0f1683c3) No inactive management server node found 2025-06-29 02:42:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b639ed56]) (logid:0f1683c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:51,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7055b47b]) (logid:1a505c41) No inactive management server node found 2025-06-29 02:42:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7055b47b]) (logid:1a505c41) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ce92b09]) (logid:d6318f76) No inactive management server node found 2025-06-29 02:42:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ce92b09]) (logid:d6318f76) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:53,464 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5771312d]) (logid:c4c100d7) HA health check task is running... 2025-06-29 02:42:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fd5fe4]) (logid:c1bc3bfe) No inactive management server node found 2025-06-29 02:42:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fd5fe4]) (logid:c1bc3bfe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:55,052 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7850: Seq 7-7850: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:42:55,062 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7850 2025-06-29 02:42:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-143ec396]) (logid:0b9aed05) No inactive management server node found 2025-06-29 02:42:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-143ec396]) (logid:0b9aed05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-30eb6787]) (logid:42516dfb) Enabled non-edge zones available for scan: 1 2025-06-29 02:42:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-30eb6787]) (logid:42516dfb) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:42:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-30eb6787]) (logid:42516dfb) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:42:56,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b04002a9]) (logid:05fa5f61) Enabled non-edge zones available for scan: 1 2025-06-29 02:42:56,896 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b04002a9]) (logid:05fa5f61) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:42:57,465 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-bda1d642]) (logid:bb600da0) HA health check task is running... 2025-06-29 02:42:57,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b37c3d7b]) (logid:12b58a0a) No inactive management server node found 2025-06-29 02:42:57,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b37c3d7b]) (logid:12b58a0a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:42:57,861 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-dddbd283]) (logid:aa8daf9a) AutoScaling Monitor is running... 2025-06-29 02:42:57,956 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-bc4d2c47]) (logid:906053c4) Starting to remove all volume_stats rows older than [Sat Jun 28 14:42:57 CEST 2025]. 2025-06-29 02:42:57,956 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-bc4d2c47]) (logid:906053c4) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:42:57 CEST 2025]. 2025-06-29 02:42:58,140 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:42:58,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:42:58,143 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) Seq 2-3048092522799696605: Routed from 90520738888109 2025-06-29 02:42:58,196 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) Seq 1-3048092522799696605: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:42:58,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:42:58,231 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) Seq 2-1612007191621929875: Routed from 90520738888109 2025-06-29 02:42:58,277 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-52464110]) (logid:5a0d4992) Seq 2-1612007191621929875: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:42:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f305f5b3]) (logid:b8e1f9eb) No inactive management server node found 2025-06-29 02:42:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f305f5b3]) (logid:b8e1f9eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a9e56d9]) (logid:51301903) No inactive management server node found 2025-06-29 02:43:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a9e56d9]) (logid:51301903) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:01,466 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-55c1b418]) (logid:da1a5e79) HA health check task is running... 2025-06-29 02:43:02,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cccecdd]) (logid:b9c3779e) No inactive management server node found 2025-06-29 02:43:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cccecdd]) (logid:b9c3779e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:03,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ca5e03b]) (logid:9323c614) No inactive management server node found 2025-06-29 02:43:03,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ca5e03b]) (logid:9323c614) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:05,052 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7851: Seq 7-7851: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:43:05,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7851 2025-06-29 02:43:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1569e5a9]) (logid:1fa35c4a) No inactive management server node found 2025-06-29 02:43:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1569e5a9]) (logid:1fa35c4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:05,468 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b8f32289]) (logid:813f6ff9) HA health check task is running... 2025-06-29 02:43:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daec8433]) (logid:ec8e26f1) No inactive management server node found 2025-06-29 02:43:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daec8433]) (logid:ec8e26f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ff56216]) (logid:f5529c2f) No inactive management server node found 2025-06-29 02:43:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ff56216]) (logid:f5529c2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:09,469 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ceea372b]) (logid:8d4d3828) HA health check task is running... 2025-06-29 02:43:09,839 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ad8d87e]) (logid:d9c60e0c) No inactive management server node found 2025-06-29 02:43:09,839 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ad8d87e]) (logid:d9c60e0c) Peer scan is finished. profiler: Done. Duration: 8ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 7ms 2025-06-29 02:43:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fa6807ff]) (logid:c93214e1) Found 0 routers to update status. 2025-06-29 02:43:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fa6807ff]) (logid:c93214e1) Found 0 VPC's to update Redundant State. 2025-06-29 02:43:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fa6807ff]) (logid:c93214e1) Found 0 networks to update RvR status. 2025-06-29 02:43:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43030a99]) (logid:469303bb) No inactive management server node found 2025-06-29 02:43:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43030a99]) (logid:469303bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce63c7de]) (logid:60884d4d) No inactive management server node found 2025-06-29 02:43:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce63c7de]) (logid:60884d4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:12,850 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-355757fe]) (logid:4d29f055) Starting to remove all vm_stats rows older than [Sat Jun 28 14:43:12 CEST 2025]. 2025-06-29 02:43:12,866 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-355757fe]) (logid:4d29f055) Removed a total of [0] vm_stats rows older than [Sat Jun 28 14:43:12 CEST 2025]. 2025-06-29 02:43:13,471 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3db6c3f4]) (logid:28d2a2a7) HA health check task is running... 2025-06-29 02:43:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1de6306]) (logid:6b0d7f16) No inactive management server node found 2025-06-29 02:43:14,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1de6306]) (logid:6b0d7f16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:15,053 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7852: Seq 7-7852: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:43:15,062 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7852 2025-06-29 02:43:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b312b3c]) (logid:56fdf3ac) No inactive management server node found 2025-06-29 02:43:15,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b312b3c]) (logid:56fdf3ac) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:43:17,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-20fe6ae7]) (logid:cf33d554) Resetting hosts suitable for reconnect 2025-06-29 02:43:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-20fe6ae7]) (logid:cf33d554) Completed resetting hosts suitable for reconnect 2025-06-29 02:43:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-20fe6ae7]) (logid:cf33d554) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:43:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-20fe6ae7]) (logid:cf33d554) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:43:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-20fe6ae7]) (logid:cf33d554) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:43:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-20fe6ae7]) (logid:cf33d554) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:43:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2ff88da]) (logid:3a09d6ad) No inactive management server node found 2025-06-29 02:43:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2ff88da]) (logid:3a09d6ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:17,472 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7bf699e8]) (logid:9752e9f2) HA health check task is running... 2025-06-29 02:43:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1675f622]) (logid:75c6c2d8) No inactive management server node found 2025-06-29 02:43:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1675f622]) (logid:75c6c2d8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dccd089]) (logid:db89b382) No inactive management server node found 2025-06-29 02:43:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dccd089]) (logid:db89b382) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:21,473 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-46790d1a]) (logid:c0dde152) HA health check task is running... 2025-06-29 02:43:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eac0f370]) (logid:5cf6c9e9) No inactive management server node found 2025-06-29 02:43:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eac0f370]) (logid:5cf6c9e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b5f3b0c]) (logid:7c6783d1) No inactive management server node found 2025-06-29 02:43:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b5f3b0c]) (logid:7c6783d1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f40b0616]) (logid:fbc1ba73) No inactive management server node found 2025-06-29 02:43:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f40b0616]) (logid:fbc1ba73) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:25,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7853: Seq 7-7853: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:43:25,064 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7853 2025-06-29 02:43:25,474 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-66230f40]) (logid:3c935bdf) HA health check task is running... 2025-06-29 02:43:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d464013]) (logid:eab44715) No inactive management server node found 2025-06-29 02:43:26,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d464013]) (logid:eab44715) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2130d613]) (logid:d8c61a37) Enabled non-edge zones available for scan: 1 2025-06-29 02:43:26,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-2130d613]) (logid:d8c61a37) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:43:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2130d613]) (logid:d8c61a37) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:43:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cc590d91]) (logid:7d9818af) Enabled non-edge zones available for scan: 1 2025-06-29 02:43:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cc590d91]) (logid:7d9818af) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:43:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af4512ae]) (logid:45afce4d) No inactive management server node found 2025-06-29 02:43:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af4512ae]) (logid:45afce4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:29,262 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:43:29,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:43:29,265 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) Seq 2-3048092522799696606: Routed from 90520738888109 2025-06-29 02:43:29,308 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) Seq 1-3048092522799696606: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:43:29,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:43:29,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) Seq 2-1612007191621929876: Routed from 90520738888109 2025-06-29 02:43:29,317 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-f1035e9e]) (logid:eeabb2bc) Seq 2-1612007191621929876: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:43:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3528d21]) (logid:df07119f) No inactive management server node found 2025-06-29 02:43:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3528d21]) (logid:df07119f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:29,475 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-774eb3b6]) (logid:a50a4878) HA health check task is running... 2025-06-29 02:43:30,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22d79e35]) (logid:fc72a48e) No inactive management server node found 2025-06-29 02:43:30,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22d79e35]) (logid:fc72a48e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf9448a6]) (logid:d3041bd9) No inactive management server node found 2025-06-29 02:43:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf9448a6]) (logid:d3041bd9) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 2ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:43:32,682 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) StorageCollector is running... 2025-06-29 02:43:32,689 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:43:32,689 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:43:32,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:43:32,690 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-6717118844223095460: Routed from 90520738888109 2025-06-29 02:43:32,745 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 4-6717118844223095460: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:43:32,750 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:43:32,750 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:43:32,750 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:43:32,751 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-1612007191621929877: Routed from 90520738888109 2025-06-29 02:43:32,803 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-1612007191621929877: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:43:32,814 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:43:32,814 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:43:32,814 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:43:32,815 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-3048092522799696607: Routed from 90520738888109 2025-06-29 02:43:32,861 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 1-3048092522799696607: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:43:32,872 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:43:32,872 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:43:32,872 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:43:32,873 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-3048092522799696608: Routed from 90520738888109 2025-06-29 02:43:33,477 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-bf116c07]) (logid:d144940f) HA health check task is running... 2025-06-29 02:43:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b93ff0c8]) (logid:3f04847a) No inactive management server node found 2025-06-29 02:43:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b93ff0c8]) (logid:3f04847a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:33,964 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 1-3048092522799696608: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:43:33,966 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:43:33,966 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:43:33,966 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:43:33,967 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-1612007191621929878: Routed from 90520738888109 2025-06-29 02:43:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:43:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:43:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:43:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:43:35,042 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b3fa96b1]) (logid:3beb4581) Seq 2-1612007191621929878: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:43:35,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7854: Seq 7-7854: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:43:35,063 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7854 2025-06-29 02:43:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b4bf859]) (logid:1ed2e693) No inactive management server node found 2025-06-29 02:43:35,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b4bf859]) (logid:1ed2e693) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b4813e]) (logid:68e26cca) No inactive management server node found 2025-06-29 02:43:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b4813e]) (logid:68e26cca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:37,478 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9aad0922]) (logid:25b95b70) HA health check task is running... 2025-06-29 02:43:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3df678bd]) (logid:4c30fa32) No inactive management server node found 2025-06-29 02:43:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3df678bd]) (logid:4c30fa32) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59e8e040]) (logid:aa1e87fa) No inactive management server node found 2025-06-29 02:43:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59e8e040]) (logid:aa1e87fa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:40,815 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-655299e1]) (logid:7f71bc83) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 02:43:40,815 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-655299e1]) (logid:7f71bc83) Snapshot scheduler is being called at [2025-06-29T00:43:40+0000]. 2025-06-29 02:43:40,816 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-655299e1]) (logid:7f71bc83) There are [0] scheduled snapshots to be executed at [2025-06-29T00:43:40+0000]. 2025-06-29 02:43:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-d8973407]) (logid:7857e223) External devices are not used. Skipping external device usage collection 2025-06-29 02:43:40,978 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-11b8795c]) (logid:d5088569) Backup backup.poll is being called at 2025-06-29T00:43:40+0000 2025-06-29 02:43:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-6853b317]) (logid:1b3286ac) Found 0 running routers. 2025-06-29 02:43:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5a5ae8a0]) (logid:e739c185) Found 0 routers to update status. 2025-06-29 02:43:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5a5ae8a0]) (logid:e739c185) Found 0 VPC's to update Redundant State. 2025-06-29 02:43:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5a5ae8a0]) (logid:e739c185) Found 0 networks to update RvR status. 2025-06-29 02:43:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-8963f042]) (logid:4a86f94d) DbCollector is running... 2025-06-29 02:43:41,276 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-5534b876]) (logid:52ef32a7) Started resource counters recalculation periodic task. 2025-06-29 02:43:41,279 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-5534b876]) (logid:52ef32a7) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 02:43:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfc34157]) (logid:2f1ed80b) No inactive management server node found 2025-06-29 02:43:41,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfc34157]) (logid:2f1ed80b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:41,479 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3252fd0c]) (logid:71f7a9d5) HA health check task is running... 2025-06-29 02:43:41,593 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-5534b876]) (logid:52ef32a7) Finished resource counters recalculation periodic task. 2025-06-29 02:43:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) ManagementServerCollector is running... 2025-06-29 02:43:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) DB properties were already loaded 2025-06-29 02:43:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:43:42,168 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Execution of process [4061847] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:43:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Exit value of process [4061847] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:43:42,168 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Process [4061847] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:43:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) usage install: null 2025-06-29 02:43:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Sessions active: 0 2025-06-29 02:43:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:43:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061850] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:43:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:43:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061855] for command [/bin/bash -c uname -r ]. 2025-06-29 02:43:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:43:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061856] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:43:42,186 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) system memory from /proc: 4887515136 2025-06-29 02:43:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:43:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061861] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:43:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) free memory from /proc: 526098432 2025-06-29 02:43:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:43:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061866] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:43:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) used memory from /proc: 755852 2025-06-29 02:43:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:43:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061867] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:43:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:43:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061868] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:43:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:43:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061869] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:43:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:43:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061876] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:43:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:43:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Successfully executed process [4061877] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:43:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail buffersdirect.count, 120 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail buffersdirect.used, 657986 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail buffersmapped.count, 7 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail buffersmapped.used, 798602 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail gcPS-Scavenge.count, 476 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail gcPS-Scavenge.time, 3501 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail jvmuptime, 41762327 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memoryheap.committed, 350224384 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memoryheap.init, 77594624 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memoryheap.usage, 0.10333997081577434 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorynon-heap.max, -1 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorynon-heap.usage, -2.20103472E8 2025-06-29 02:43:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorynon-heap.used, 220103472 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17537719108237804 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.PS-Eden-Space.usage, 0.03288930604742075 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.PS-Old-Gen.usage, 0.11558773525465718 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorypools.PS-Survivor-Space.usage, 0.7226619720458984 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorytotal.committed, 575078400 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorytotal.init, 85262336 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorytotal.max, 1908932607 2025-06-29 02:43:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail memorytotal.used, 417426296 2025-06-29 02:43:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail threadsdeadlocks, [] 2025-06-29 02:43:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail threadsnew.count, 0 2025-06-29 02:43:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:43:42,255 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-f73a8d6e]) (logid:930fd2df) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:43:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":153719720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41762245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":416626096,"totalCommitted":575078400,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197269040,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":526098432,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.26,0.23],"systemCyclesUsage":[66361139,53801914,4587435860],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:43:42,256 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1391, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:43:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":153719720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41762245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":416626096,"totalCommitted":575078400,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197269040,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":526098432,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.26,0.23],"systemCyclesUsage":[66361139,53801914,4587435860],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:43:42,256 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:43:42,262 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-312:[ctx-00bed024]) (logid:d4487174) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:43:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":153719720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41762245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":416626096,"totalCommitted":575078400,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197269040,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":526098432,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.26,0.23],"systemCyclesUsage":[66361139,53801914,4587435860],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:43:42,262 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 02:43:42,262 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1391, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:43:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":153719720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41762245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":416626096,"totalCommitted":575078400,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197269040,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":526098432,"systemMemoryUsed":755852,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.26,0.23],"systemCyclesUsage":[66361139,53801914,4587435860],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:43:42,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-677124d6]) (logid:d90dc6a4) No inactive management server node found 2025-06-29 02:43:42,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-677124d6]) (logid:d90dc6a4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2c996c9]) (logid:98c04149) No inactive management server node found 2025-06-29 02:43:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2c996c9]) (logid:98c04149) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:45,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7855: Seq 7-7855: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:43:45,064 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7855 2025-06-29 02:43:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:43:45,481 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ae5911c2]) (logid:b88a212a) HA health check task is running... 2025-06-29 02:43:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:43:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51900d1b]) (logid:a3fc8942) No inactive management server node found 2025-06-29 02:43:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51900d1b]) (logid:a3fc8942) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-35770fd8]) (logid:a0636cb2) VM scheduler.poll is being called at 2025-06-29T00:44:00+0000 2025-06-29 02:43:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-35770fd8]) (logid:a0636cb2) Got 0 scheduled jobs to be executed at 2025-06-29T00:44:00+0000 2025-06-29 02:43:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-35770fd8]) (logid:a0636cb2) Cleaned up 0 VM scheduled job entries 2025-06-29 02:43:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-29a66e7f]) (logid:4358a718) VM Operation Thread Running 2025-06-29 02:43:46,175 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-2:[ctx-675d2057]) (logid:ab4d2681) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 02:43:46,329 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-dcde41bc]) (logid:ba79ba51) ClusterDRS.poll is being called at 2025-06-29T00:44:00+0000 2025-06-29 02:43:46,336 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-dcde41bc]) (logid:ba79ba51) Removed 0 old drs migration plans 2025-06-29 02:43:47,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:43:47,237 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:43:47,239 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:43:47,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:43:47,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e7205de]) (logid:89851dbc) No inactive management server node found 2025-06-29 02:43:47,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e7205de]) (logid:89851dbc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd0e96ea]) (logid:6f6e3868) No inactive management server node found 2025-06-29 02:43:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd0e96ea]) (logid:6f6e3868) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:49,482 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a5da64b0]) (logid:973e7a8f) HA health check task is running... 2025-06-29 02:43:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32438c6c]) (logid:aae57b21) No inactive management server node found 2025-06-29 02:43:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32438c6c]) (logid:aae57b21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e6696a9]) (logid:32ffdf29) No inactive management server node found 2025-06-29 02:43:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e6696a9]) (logid:32ffdf29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c314cd4e]) (logid:5bf830b1) No inactive management server node found 2025-06-29 02:43:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c314cd4e]) (logid:5bf830b1) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:43:53,483 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d28e7992]) (logid:200d0e2a) HA health check task is running... 2025-06-29 02:43:55,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7857: Seq 7-7857: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:43:55,997 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7857 2025-06-29 02:43:56,006 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37af1842]) (logid:9e082203) No inactive management server node found 2025-06-29 02:43:56,006 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37af1842]) (logid:9e082203) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3646f0d9]) (logid:b5f11505) No inactive management server node found 2025-06-29 02:43:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3646f0d9]) (logid:b5f11505) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-12cc88e3]) (logid:3f1c968e) Enabled non-edge zones available for scan: 1 2025-06-29 02:43:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-12cc88e3]) (logid:3f1c968e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:43:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-12cc88e3]) (logid:3f1c968e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:43:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cd614a9a]) (logid:67a84c31) Enabled non-edge zones available for scan: 1 2025-06-29 02:43:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cd614a9a]) (logid:67a84c31) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:43:57,485 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-384b5af4]) (logid:72789203) HA health check task is running... 2025-06-29 02:43:57,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e14f9d75]) (logid:c196efe3) No inactive management server node found 2025-06-29 02:43:57,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e14f9d75]) (logid:c196efe3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:43:57,864 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-6d67564b]) (logid:79bded92) AutoScaling Monitor is running... 2025-06-29 02:43:57,958 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-c5cbeb25]) (logid:3f47b7df) Starting to remove all volume_stats rows older than [Sat Jun 28 14:43:57 CEST 2025]. 2025-06-29 02:43:57,959 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-c5cbeb25]) (logid:3f47b7df) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:43:57 CEST 2025]. 2025-06-29 02:43:58,331 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:43:58,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:43:58,337 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) Seq 2-3048092522799696609: Routed from 90520738888109 2025-06-29 02:43:58,388 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) Seq 1-3048092522799696609: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:43:58,435 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:43:58,435 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) Seq 2-1612007191621929879: Routed from 90520738888109 2025-06-29 02:43:58,483 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-312a9e40]) (logid:adbdcb6e) Seq 2-1612007191621929879: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:43:59,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32fc4232]) (logid:e0e9b24e) No inactive management server node found 2025-06-29 02:43:59,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32fc4232]) (logid:e0e9b24e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed78651e]) (logid:4bdc9556) No inactive management server node found 2025-06-29 02:44:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed78651e]) (logid:4bdc9556) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:01,486 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0a489915]) (logid:7c3244ae) HA health check task is running... 2025-06-29 02:44:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f387a96e]) (logid:4b1fd743) No inactive management server node found 2025-06-29 02:44:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f387a96e]) (logid:4b1fd743) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:03,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d0ef5c8]) (logid:8948602a) No inactive management server node found 2025-06-29 02:44:03,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d0ef5c8]) (logid:8948602a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:05,014 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7858: Seq 7-7858: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:44:05,024 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7858 2025-06-29 02:44:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e92aac89]) (logid:43e64ea8) No inactive management server node found 2025-06-29 02:44:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e92aac89]) (logid:43e64ea8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:05,487 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c6ec9549]) (logid:1140e640) HA health check task is running... 2025-06-29 02:44:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8fd649e]) (logid:7bfc72f1) No inactive management server node found 2025-06-29 02:44:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8fd649e]) (logid:7bfc72f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:08,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dd844c7]) (logid:3ceab063) No inactive management server node found 2025-06-29 02:44:08,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dd844c7]) (logid:3ceab063) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:09,489 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dbdc5ac1]) (logid:ced27f41) HA health check task is running... 2025-06-29 02:44:09,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-382a988e]) (logid:8f84aeae) No inactive management server node found 2025-06-29 02:44:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-382a988e]) (logid:8f84aeae) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9265e107]) (logid:b93f41b2) Found 0 routers to update status. 2025-06-29 02:44:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9265e107]) (logid:b93f41b2) Found 0 VPC's to update Redundant State. 2025-06-29 02:44:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9265e107]) (logid:b93f41b2) Found 0 networks to update RvR status. 2025-06-29 02:44:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Running Capacity Checker ... 2025-06-29 02:44:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) recalculating system capacity 2025-06-29 02:44:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Executing cpu/ram capacity update 2025-06-29 02:44:11,069 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:11,070 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:11,076 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:11,079 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 02:44:11,079 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 02:44:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 02:44:11,102 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:11,103 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:11,107 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:11,108 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 02:44:11,109 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 02:44:11,109 DEBUG [c.c.c.CapacityManagerImpl] (pool-306-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 02:44:11,119 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Done executing cpu/ram capacity update 2025-06-29 02:44:11,119 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Executing storage capacity update 2025-06-29 02:44:11,125 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:44:11,125 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 02:44:11,125 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 02:44:11,127 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 02:44:11,132 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:44:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 02:44:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 02:44:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 02:44:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:44:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 02:44:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 02:44:11,142 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 02:44:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:44:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 02:44:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 02:44:11,160 DEBUG [c.c.s.StorageManagerImpl] (pool-307-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 02:44:11,160 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Done executing storage capacity update 2025-06-29 02:44:11,160 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Executing capacity updates for public ip and Vlans 2025-06-29 02:44:11,174 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Done capacity updates for public ip and Vlans 2025-06-29 02:44:11,174 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Executing capacity updates for private ip 2025-06-29 02:44:11,177 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Done executing capacity updates for private ip 2025-06-29 02:44:11,177 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Done recalculating system capacity 2025-06-29 02:44:11,201 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-f5450494]) (logid:7e435c49) Done running Capacity Checker ... 2025-06-29 02:44:11,257 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-248f39b9]) (logid:7cc5238b) Checking if any host reservation can be released ... 2025-06-29 02:44:11,261 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-248f39b9]) (logid:7cc5238b) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:11,264 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-248f39b9]) (logid:7cc5238b) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:11,264 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-248f39b9]) (logid:7cc5238b) Done running HostReservationReleaseChecker ... 2025-06-29 02:44:11,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994d5912]) (logid:9188dacd) No inactive management server node found 2025-06-29 02:44:11,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994d5912]) (logid:9188dacd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:12,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92947463]) (logid:b0a9a16a) No inactive management server node found 2025-06-29 02:44:12,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92947463]) (logid:b0a9a16a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:12,868 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-2165e9c0]) (logid:c0765d7d) Starting to remove all vm_stats rows older than [Sat Jun 28 14:44:12 CEST 2025]. 2025-06-29 02:44:12,883 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-2165e9c0]) (logid:c0765d7d) Removed a total of [0] vm_stats rows older than [Sat Jun 28 14:44:12 CEST 2025]. 2025-06-29 02:44:13,490 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8fc5e503]) (logid:66e2c065) HA health check task is running... 2025-06-29 02:44:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44153d11]) (logid:752933e5) No inactive management server node found 2025-06-29 02:44:14,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44153d11]) (logid:752933e5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:15,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7859: Seq 7-7859: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:44:15,065 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7859 2025-06-29 02:44:15,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da44f725]) (logid:ef37424c) No inactive management server node found 2025-06-29 02:44:15,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da44f725]) (logid:ef37424c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:44:17,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd42a6e8]) (logid:916f4056) No inactive management server node found 2025-06-29 02:44:17,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd42a6e8]) (logid:916f4056) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:17,492 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b9e44cec]) (logid:294bac90) HA health check task is running... 2025-06-29 02:44:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d090a21]) (logid:f4d86f97) No inactive management server node found 2025-06-29 02:44:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d090a21]) (logid:f4d86f97) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:20,056 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7860: Seq 7-7860: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:44:20,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7860 2025-06-29 02:44:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4a9b958]) (logid:e835a0a8) No inactive management server node found 2025-06-29 02:44:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4a9b958]) (logid:e835a0a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:21,493 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-368f5c37]) (logid:fec9040b) HA health check task is running... 2025-06-29 02:44:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3902984a]) (logid:4583934a) No inactive management server node found 2025-06-29 02:44:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3902984a]) (logid:4583934a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfd50f7b]) (logid:3e6901fc) No inactive management server node found 2025-06-29 02:44:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfd50f7b]) (logid:3e6901fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:24,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-233c5da5]) (logid:c8193ef3) No inactive management server node found 2025-06-29 02:44:24,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-233c5da5]) (logid:c8193ef3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:25,495 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-31c2bfa4]) (logid:9a985a8b) HA health check task is running... 2025-06-29 02:44:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4c42594]) (logid:65d67cc6) No inactive management server node found 2025-06-29 02:44:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4c42594]) (logid:65d67cc6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ee9f9bb3]) (logid:ade20d1d) Enabled non-edge zones available for scan: 1 2025-06-29 02:44:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ee9f9bb3]) (logid:ade20d1d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:44:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ee9f9bb3]) (logid:ade20d1d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:44:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b0fdf9ef]) (logid:cb2dd866) Enabled non-edge zones available for scan: 1 2025-06-29 02:44:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b0fdf9ef]) (logid:cb2dd866) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:44:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-120a5ada]) (logid:b67945d7) No inactive management server node found 2025-06-29 02:44:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-120a5ada]) (logid:b67945d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:29,321 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:44:29,323 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:44:29,325 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) Seq 2-3048092522799696610: Routed from 90520738888109 2025-06-29 02:44:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b01a97]) (logid:423c219a) No inactive management server node found 2025-06-29 02:44:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b01a97]) (logid:423c219a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:29,369 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) Seq 1-3048092522799696610: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:44:29,371 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:44:29,371 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) Seq 2-1612007191621929880: Routed from 90520738888109 2025-06-29 02:44:29,374 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-fcc47760]) (logid:7bd9790d) Seq 2-1612007191621929880: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:44:29,496 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d497ee5f]) (logid:3e9f2fb9) HA health check task is running... 2025-06-29 02:44:30,056 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7861: Seq 7-7861: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:44:30,065 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7861 2025-06-29 02:44:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e251af3]) (logid:6f1c72aa) No inactive management server node found 2025-06-29 02:44:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e251af3]) (logid:6f1c72aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18652aa6]) (logid:bd25e3ba) No inactive management server node found 2025-06-29 02:44:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18652aa6]) (logid:bd25e3ba) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:33,497 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c3016e62]) (logid:1b6c05a6) HA health check task is running... 2025-06-29 02:44:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f15768b]) (logid:46b132b0) No inactive management server node found 2025-06-29 02:44:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f15768b]) (logid:46b132b0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:33,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:44:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:44:35,043 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) StorageCollector is running... 2025-06-29 02:44:35,048 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:44:35,048 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:44:35,048 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:44:35,049 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-6717118844223095461: Routed from 90520738888109 2025-06-29 02:44:35,112 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 4-6717118844223095461: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:44:35,116 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:44:35,116 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:44:35,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:44:35,124 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-1612007191621929881: Routed from 90520738888109 2025-06-29 02:44:35,123 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 02:44:35,180 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-1612007191621929881: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:44:35,204 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:44:35,204 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:44:35,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:44:35,205 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-1612007191621929882: Routed from 90520738888109 2025-06-29 02:44:35,300 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-1612007191621929882: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:44:35,311 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:44:35,311 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:44:35,312 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:44:35,313 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-3048092522799696611: Routed from 90520738888109 2025-06-29 02:44:35,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-656e6643]) (logid:42bb0708) No inactive management server node found 2025-06-29 02:44:35,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-656e6643]) (logid:42bb0708) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:44:36,363 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 1-3048092522799696611: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:44:36,365 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:44:36,365 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:44:36,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:44:36,366 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-1612007191621929883: Routed from 90520738888109 2025-06-29 02:44:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-932c86e8]) (logid:78bc4da2) No inactive management server node found 2025-06-29 02:44:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-932c86e8]) (logid:78bc4da2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:37,424 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609db12c]) (logid:404aaeab) Seq 2-1612007191621929883: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:44:37,498 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8bc2e1bd]) (logid:2c36ce13) HA health check task is running... 2025-06-29 02:44:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b72d2b6]) (logid:294a95dd) No inactive management server node found 2025-06-29 02:44:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b72d2b6]) (logid:294a95dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77990971]) (logid:e55ebb37) No inactive management server node found 2025-06-29 02:44:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77990971]) (logid:e55ebb37) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:40,056 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7862: Seq 7-7862: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:44:40,066 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7862 2025-06-29 02:44:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2d917832]) (logid:4155a255) Found 0 routers to update status. 2025-06-29 02:44:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2d917832]) (logid:4155a255) Found 0 VPC's to update Redundant State. 2025-06-29 02:44:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2d917832]) (logid:4155a255) Found 0 networks to update RvR status. 2025-06-29 02:44:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-17403b25]) (logid:6c6aed67) DbCollector is running... 2025-06-29 02:44:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8313bd7c]) (logid:49cfa0b8) No inactive management server node found 2025-06-29 02:44:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8313bd7c]) (logid:49cfa0b8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:44:41,500 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-754a167c]) (logid:80e32c43) HA health check task is running... 2025-06-29 02:44:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) ManagementServerCollector is running... 2025-06-29 02:44:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) DB properties were already loaded 2025-06-29 02:44:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:44:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Execution of process [4061914] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:44:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Exit value of process [4061914] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:44:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Process [4061914] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:44:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) usage install: null 2025-06-29 02:44:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Sessions active: 0 2025-06-29 02:44:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:44:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061917] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:44:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:44:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061922] for command [/bin/bash -c uname -r ]. 2025-06-29 02:44:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:44:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061923] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:44:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) system memory from /proc: 4887515136 2025-06-29 02:44:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:44:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061928] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:44:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) free memory from /proc: 517689344 2025-06-29 02:44:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:44:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061933] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:44:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) used memory from /proc: 755980 2025-06-29 02:44:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:44:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061934] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:44:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:44:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061935] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:44:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:44:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061936] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:44:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:44:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061943] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:44:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:44:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Successfully executed process [4061944] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail buffersdirect.count, 120 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail buffersdirect.used, 657986 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail buffersmapped.count, 7 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail buffersmapped.used, 798602 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail gcPS-Scavenge.count, 478 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail gcPS-Scavenge.time, 3514 2025-06-29 02:44:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail jvmuptime, 41822321 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memoryheap.committed, 350748672 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memoryheap.init, 77594624 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memoryheap.usage, 0.09337557923888741 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorynon-heap.max, -1 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorynon-heap.usage, -2.20103472E8 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorynon-heap.used, 220103472 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17537719108237804 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.PS-Eden-Space.usage, 0.00515351129048511 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.PS-Old-Gen.usage, 0.11657860671063999 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorypools.PS-Survivor-Space.usage, 0.6197961171468099 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorytotal.committed, 575602688 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorytotal.init, 85262336 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorytotal.max, 1908932607 2025-06-29 02:44:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail memorytotal.used, 398406312 2025-06-29 02:44:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail threadsdeadlocks, [] 2025-06-29 02:44:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail threadsnew.count, 0 2025-06-29 02:44:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:44:42,248 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-659c3e86]) (logid:e5fe0c27) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:44:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":173266616,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":397596584,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":178247688,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517689344,"systemMemoryUsed":755980,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.24,0.23],"systemCyclesUsage":[66361469,53802235,4587458951],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:44:42,248 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1393, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:44:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":173266616,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":397596584,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":178247688,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517689344,"systemMemoryUsed":755980,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.24,0.23],"systemCyclesUsage":[66361469,53802235,4587458951],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:44:42,248 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:44:42,253 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-312:[ctx-5607eff8]) (logid:6f58bfea) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:44:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":173266616,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":397596584,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":178247688,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517689344,"systemMemoryUsed":755980,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.24,0.23],"systemCyclesUsage":[66361469,53802235,4587458951],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:44:42,253 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:44:42,253 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1393, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:44:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":173266616,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":397596584,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":178247688,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517689344,"systemMemoryUsed":755980,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.24,0.23],"systemCyclesUsage":[66361469,53802235,4587458951],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:44:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99c80690]) (logid:e825486c) No inactive management server node found 2025-06-29 02:44:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99c80690]) (logid:e825486c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:44,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c38d9c9c]) (logid:12c9b5bd) No inactive management server node found 2025-06-29 02:44:44,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c38d9c9c]) (logid:12c9b5bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:44:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:44:45,501 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-596a8238]) (logid:d51e6af3) HA health check task is running... 2025-06-29 02:44:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3826106b]) (logid:560f194a) No inactive management server node found 2025-06-29 02:44:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3826106b]) (logid:560f194a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-eea88ec5]) (logid:6f03b9e9) VM scheduler.poll is being called at 2025-06-29T00:45:00+0000 2025-06-29 02:44:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-eea88ec5]) (logid:6f03b9e9) Got 0 scheduled jobs to be executed at 2025-06-29T00:45:00+0000 2025-06-29 02:44:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-eea88ec5]) (logid:6f03b9e9) Cleaned up 0 VM scheduled job entries 2025-06-29 02:44:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-4527158a]) (logid:c5a0d1b4) VM Operation Thread Running 2025-06-29 02:44:46,329 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-fb342fca]) (logid:527afa95) ClusterDRS.poll is being called at 2025-06-29T00:45:00+0000 2025-06-29 02:44:46,336 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-fb342fca]) (logid:527afa95) Removed 0 old drs migration plans 2025-06-29 02:44:47,051 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84e8e494]) (logid:0b0d9588) Resetting hosts suitable for reconnect 2025-06-29 02:44:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84e8e494]) (logid:0b0d9588) Completed resetting hosts suitable for reconnect 2025-06-29 02:44:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84e8e494]) (logid:0b0d9588) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:44:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84e8e494]) (logid:0b0d9588) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:44:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84e8e494]) (logid:0b0d9588) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:44:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84e8e494]) (logid:0b0d9588) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:44:47,233 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:47,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:47,236 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:44:47,238 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:44:47,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-031c880f]) (logid:ca0bc0e4) No inactive management server node found 2025-06-29 02:44:47,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-031c880f]) (logid:ca0bc0e4) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6be47645]) (logid:d96f3ab6) No inactive management server node found 2025-06-29 02:44:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6be47645]) (logid:d96f3ab6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:49,502 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0e706efb]) (logid:76b00f07) HA health check task is running... 2025-06-29 02:44:50,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7864: Seq 7-7864: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:44:50,078 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7864 2025-06-29 02:44:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4a8ea14]) (logid:42297ea9) No inactive management server node found 2025-06-29 02:44:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4a8ea14]) (logid:42297ea9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:51,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f771c492]) (logid:f49a8783) No inactive management server node found 2025-06-29 02:44:51,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f771c492]) (logid:f49a8783) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:53,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad214787]) (logid:032de001) No inactive management server node found 2025-06-29 02:44:53,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad214787]) (logid:032de001) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:53,503 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b6947074]) (logid:dbc0b7e6) HA health check task is running... 2025-06-29 02:44:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e26f9f1]) (logid:c947bdc7) No inactive management server node found 2025-06-29 02:44:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e26f9f1]) (logid:c947bdc7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:56,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8139c51]) (logid:2723313b) No inactive management server node found 2025-06-29 02:44:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8139c51]) (logid:2723313b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:44:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-09024597]) (logid:3d8073ec) Enabled non-edge zones available for scan: 1 2025-06-29 02:44:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-09024597]) (logid:3d8073ec) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:44:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-09024597]) (logid:3d8073ec) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:44:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0f345df7]) (logid:dc20f8af) Enabled non-edge zones available for scan: 1 2025-06-29 02:44:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0f345df7]) (logid:dc20f8af) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:44:57,505 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-91f41d99]) (logid:b5ff54ef) HA health check task is running... 2025-06-29 02:44:57,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02aa6e5f]) (logid:11f7e11d) No inactive management server node found 2025-06-29 02:44:57,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02aa6e5f]) (logid:11f7e11d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:44:57,865 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-53141d60]) (logid:b689b197) AutoScaling Monitor is running... 2025-06-29 02:44:57,961 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-3ca7329b]) (logid:f0baa2eb) Starting to remove all volume_stats rows older than [Sat Jun 28 14:44:57 CEST 2025]. 2025-06-29 02:44:57,962 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-3ca7329b]) (logid:f0baa2eb) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:44:57 CEST 2025]. 2025-06-29 02:44:58,541 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:44:58,545 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:44:58,546 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) Seq 2-3048092522799696612: Routed from 90520738888109 2025-06-29 02:44:58,558 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) Seq 1-3048092522799696612: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:44:58,603 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:44:58,603 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) Seq 2-1612007191621929884: Routed from 90520738888109 2025-06-29 02:44:58,652 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-ec594aca]) (logid:43e33b04) Seq 2-1612007191621929884: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:44:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d577eaf0]) (logid:62f74fd2) No inactive management server node found 2025-06-29 02:44:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d577eaf0]) (logid:62f74fd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:00,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7865: Seq 7-7865: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:00,066 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7865 2025-06-29 02:45:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebe974a9]) (logid:b39f2d16) No inactive management server node found 2025-06-29 02:45:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebe974a9]) (logid:b39f2d16) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:01,506 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c83d2a02]) (logid:84639340) HA health check task is running... 2025-06-29 02:45:02,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6f74f9a]) (logid:72a5e726) No inactive management server node found 2025-06-29 02:45:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6f74f9a]) (logid:72a5e726) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62ce9804]) (logid:2175af53) No inactive management server node found 2025-06-29 02:45:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62ce9804]) (logid:2175af53) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:05,058 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7866: Seq 7-7866: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:05,078 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7866 2025-06-29 02:45:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61ea5de4]) (logid:054a5d61) No inactive management server node found 2025-06-29 02:45:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61ea5de4]) (logid:054a5d61) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:05,507 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e2d4ea09]) (logid:2fb6873a) HA health check task is running... 2025-06-29 02:45:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3f15295]) (logid:a1d19e2f) No inactive management server node found 2025-06-29 02:45:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3f15295]) (logid:a1d19e2f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:08,338 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cd9f8da]) (logid:e5083eb2) No inactive management server node found 2025-06-29 02:45:08,338 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cd9f8da]) (logid:e5083eb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:09,508 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ec0fb3a7]) (logid:e7bbb211) HA health check task is running... 2025-06-29 02:45:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e18edbf]) (logid:6efdec8c) No inactive management server node found 2025-06-29 02:45:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e18edbf]) (logid:6efdec8c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0a451689]) (logid:35e5797a) Found 0 routers to update status. 2025-06-29 02:45:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0a451689]) (logid:35e5797a) Found 0 VPC's to update Redundant State. 2025-06-29 02:45:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0a451689]) (logid:35e5797a) Found 0 networks to update RvR status. 2025-06-29 02:45:11,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2e84982]) (logid:ddbc1f32) No inactive management server node found 2025-06-29 02:45:11,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2e84982]) (logid:ddbc1f32) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:45:12,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1510cce3]) (logid:8945ad2e) No inactive management server node found 2025-06-29 02:45:12,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1510cce3]) (logid:8945ad2e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:12,885 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-0fc6b786]) (logid:1bf26000) Starting to remove all vm_stats rows older than [Sat Jun 28 14:45:12 CEST 2025]. 2025-06-29 02:45:12,909 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-0fc6b786]) (logid:1bf26000) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:45:12 CEST 2025]. 2025-06-29 02:45:13,510 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-68f4b213]) (logid:e5ed7fa3) HA health check task is running... 2025-06-29 02:45:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0312beb]) (logid:39ebe6d2) No inactive management server node found 2025-06-29 02:45:14,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0312beb]) (logid:39ebe6d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:15,059 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7867: Seq 7-7867: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:15,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7867 2025-06-29 02:45:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eab818d]) (logid:b538f7f7) No inactive management server node found 2025-06-29 02:45:15,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eab818d]) (logid:b538f7f7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:17,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-772f372d]) (logid:91d0aad7) No inactive management server node found 2025-06-29 02:45:17,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-772f372d]) (logid:91d0aad7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:17,512 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-375968ac]) (logid:fbbfa1e2) HA health check task is running... 2025-06-29 02:45:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06c1610a]) (logid:f1d2474d) No inactive management server node found 2025-06-29 02:45:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06c1610a]) (logid:f1d2474d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:20,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c72819e]) (logid:7fdaec11) No inactive management server node found 2025-06-29 02:45:20,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c72819e]) (logid:7fdaec11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:21,513 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4d1b9591]) (logid:1f235307) HA health check task is running... 2025-06-29 02:45:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea18b76e]) (logid:d1c4ee9a) No inactive management server node found 2025-06-29 02:45:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea18b76e]) (logid:d1c4ee9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-605ad66a]) (logid:16bc997c) No inactive management server node found 2025-06-29 02:45:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-605ad66a]) (logid:16bc997c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:24,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30531ba4]) (logid:5c46fa18) No inactive management server node found 2025-06-29 02:45:24,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30531ba4]) (logid:5c46fa18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:25,059 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7868: Seq 7-7868: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:25,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7868 2025-06-29 02:45:25,515 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-afea9d20]) (logid:cd0cf836) HA health check task is running... 2025-06-29 02:45:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3084f571]) (logid:76cc903e) No inactive management server node found 2025-06-29 02:45:26,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3084f571]) (logid:76cc903e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 2ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-82add55c]) (logid:22315b14) Enabled non-edge zones available for scan: 1 2025-06-29 02:45:26,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-82add55c]) (logid:22315b14) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2312 TB], threshold=[90.00%]. 2025-06-29 02:45:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-82add55c]) (logid:22315b14) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:45:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5f21323b]) (logid:fd675cd5) Enabled non-edge zones available for scan: 1 2025-06-29 02:45:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5f21323b]) (logid:fd675cd5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:45:27,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-583fc946]) (logid:593c558c) No inactive management server node found 2025-06-29 02:45:27,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-583fc946]) (logid:593c558c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:29,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15d48126]) (logid:7ece2d4b) No inactive management server node found 2025-06-29 02:45:29,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15d48126]) (logid:7ece2d4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:29,377 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:45:29,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:45:29,379 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) Seq 2-3048092522799696613: Routed from 90520738888109 2025-06-29 02:45:29,423 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) Seq 1-3048092522799696613: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:45:29,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:45:29,425 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) Seq 2-1612007191621929885: Routed from 90520738888109 2025-06-29 02:45:29,427 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-9472e942]) (logid:bfe16c49) Seq 2-1612007191621929885: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:45:29,516 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a813157b]) (logid:4c78852a) HA health check task is running... 2025-06-29 02:45:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2ab834d]) (logid:fe6e9652) No inactive management server node found 2025-06-29 02:45:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2ab834d]) (logid:fe6e9652) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:32,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2f6b0e1]) (logid:dc625af9) No inactive management server node found 2025-06-29 02:45:32,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2f6b0e1]) (logid:dc625af9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:33,517 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1182a59e]) (logid:654d806d) HA health check task is running... 2025-06-29 02:45:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0256d815]) (logid:7f10b37c) No inactive management server node found 2025-06-29 02:45:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0256d815]) (logid:7f10b37c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:33,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:45:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:45:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:45:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:45:35,059 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7869: Seq 7-7869: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:35,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7869 2025-06-29 02:45:35,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53b46b6c]) (logid:bbdc88e0) No inactive management server node found 2025-06-29 02:45:35,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53b46b6c]) (logid:bbdc88e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:36,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-160290df]) (logid:b0a4c1a2) No inactive management server node found 2025-06-29 02:45:36,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-160290df]) (logid:b0a4c1a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:37,424 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) StorageCollector is running... 2025-06-29 02:45:37,429 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:45:37,429 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:45:37,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:45:37,430 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-6717118844223095462: Routed from 90520738888109 2025-06-29 02:45:37,484 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 4-6717118844223095462: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:45:37,488 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:45:37,488 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:45:37,489 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:45:37,490 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-3048092522799696614: Routed from 90520738888109 2025-06-29 02:45:37,519 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f5c8bde2]) (logid:f4d56bad) HA health check task is running... 2025-06-29 02:45:37,537 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 1-3048092522799696614: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:45:37,548 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:45:37,548 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:45:37,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:45:37,549 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-1612007191621929886: Routed from 90520738888109 2025-06-29 02:45:37,601 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-1612007191621929886: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:45:37,611 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:45:37,611 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:45:37,611 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:45:37,612 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-3048092522799696615: Routed from 90520738888109 2025-06-29 02:45:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d09c135e]) (logid:835d2cd5) No inactive management server node found 2025-06-29 02:45:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d09c135e]) (logid:835d2cd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:38,663 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 1-3048092522799696615: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:45:38,665 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:45:38,665 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:45:38,665 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:45:38,666 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-1612007191621929887: Routed from 90520738888109 2025-06-29 02:45:39,723 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f8375ed6]) (logid:f31c2d59) Seq 2-1612007191621929887: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:45:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5be0b962]) (logid:f52f2833) No inactive management server node found 2025-06-29 02:45:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5be0b962]) (logid:f52f2833) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2be88114]) (logid:22c9cfe6) Found 0 routers to update status. 2025-06-29 02:45:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2be88114]) (logid:22c9cfe6) Found 0 VPC's to update Redundant State. 2025-06-29 02:45:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2be88114]) (logid:22c9cfe6) Found 0 networks to update RvR status. 2025-06-29 02:45:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-a9099fb1]) (logid:746543ad) DbCollector is running... 2025-06-29 02:45:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5dd6a7b3]) (logid:a6c44632) No inactive management server node found 2025-06-29 02:45:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5dd6a7b3]) (logid:a6c44632) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:41,520 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-96ab3104]) (logid:17b1b3f6) HA health check task is running... 2025-06-29 02:45:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) ManagementServerCollector is running... 2025-06-29 02:45:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) DB properties were already loaded 2025-06-29 02:45:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:45:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Execution of process [4061951] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:45:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Exit value of process [4061951] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:45:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Process [4061951] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:45:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) usage install: null 2025-06-29 02:45:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Sessions active: 0 2025-06-29 02:45:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:45:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061954] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:45:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:45:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061959] for command [/bin/bash -c uname -r ]. 2025-06-29 02:45:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:45:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061960] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:45:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) system memory from /proc: 4887515136 2025-06-29 02:45:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:45:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061965] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:45:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) free memory from /proc: 515399680 2025-06-29 02:45:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:45:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061970] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:45:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) used memory from /proc: 756108 2025-06-29 02:45:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:45:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061971] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:45:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:45:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061972] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:45:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:45:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061973] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:45:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:45:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061980] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:45:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:45:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Successfully executed process [4061981] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail buffersdirect.count, 120 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail buffersdirect.used, 657986 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail buffersmapped.count, 7 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail buffersmapped.used, 798602 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail gcPS-Scavenge.count, 479 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail gcPS-Scavenge.time, 3521 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail jvmuptime, 41882319 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memoryheap.committed, 350748672 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memoryheap.init, 77594624 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memoryheap.usage, 0.09250066935836009 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorynon-heap.max, -1 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorynon-heap.usage, -2.20103472E8 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorynon-heap.used, 220103472 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17537719108237804 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.PS-Eden-Space.usage, 0.0027445228466759855 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.PS-Old-Gen.usage, 0.11668159096549169 2025-06-29 02:45:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorypools.PS-Survivor-Space.usage, 0.6345188306725543 2025-06-29 02:45:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorytotal.committed, 575602688 2025-06-29 02:45:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorytotal.init, 85262336 2025-06-29 02:45:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorytotal.max, 1908932607 2025-06-29 02:45:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail memorytotal.used, 396741112 2025-06-29 02:45:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail threadsdeadlocks, [] 2025-06-29 02:45:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail threadsnew.count, 0 2025-06-29 02:45:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:45:42,246 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-75903976]) (logid:c8ffcfa2) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:45:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":174929072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41882247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":395925728,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":176577544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":515399680,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.23,0.22],"systemCyclesUsage":[66361796,53802561,4587482027],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:45:42,247 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1395, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:45:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":174929072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41882247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":395925728,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":176577544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":515399680,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.23,0.22],"systemCyclesUsage":[66361796,53802561,4587482027],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:45:42,247 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:45:42,251 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-312:[ctx-637d7679]) (logid:f8417a1a) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:45:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":174929072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41882247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":395925728,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":176577544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":515399680,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.23,0.22],"systemCyclesUsage":[66361796,53802561,4587482027],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:45:42,251 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:45:42,251 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1395, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:45:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":174929072,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41882247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":395925728,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":176577544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":515399680,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.23,0.22],"systemCyclesUsage":[66361796,53802561,4587482027],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:45:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4aed6bc9]) (logid:9ea95b81) No inactive management server node found 2025-06-29 02:45:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4aed6bc9]) (logid:9ea95b81) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:44,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f71c43c5]) (logid:48eb7696) No inactive management server node found 2025-06-29 02:45:44,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f71c43c5]) (logid:48eb7696) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:45,060 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7870: Seq 7-7870: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:45,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7870 2025-06-29 02:45:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:45:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:45:45,522 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-02fccc15]) (logid:6ce04608) HA health check task is running... 2025-06-29 02:45:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2fbfafa7]) (logid:c207a336) No inactive management server node found 2025-06-29 02:45:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2fbfafa7]) (logid:c207a336) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-24417c43]) (logid:13e9d9bc) VM scheduler.poll is being called at 2025-06-29T00:46:00+0000 2025-06-29 02:45:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-24417c43]) (logid:13e9d9bc) Got 0 scheduled jobs to be executed at 2025-06-29T00:46:00+0000 2025-06-29 02:45:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-24417c43]) (logid:13e9d9bc) Cleaned up 0 VM scheduled job entries 2025-06-29 02:45:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ead338ed]) (logid:3efac472) VM Operation Thread Running 2025-06-29 02:45:46,329 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5eed4023]) (logid:d26950b8) ClusterDRS.poll is being called at 2025-06-29T00:46:00+0000 2025-06-29 02:45:46,337 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5eed4023]) (logid:d26950b8) Removed 0 old drs migration plans 2025-06-29 02:45:47,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:45:47,244 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:45:47,245 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:45:47,247 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:45:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14409347]) (logid:e1ddae52) No inactive management server node found 2025-06-29 02:45:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14409347]) (logid:e1ddae52) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:48,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ec9b010]) (logid:46991341) No inactive management server node found 2025-06-29 02:45:48,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ec9b010]) (logid:46991341) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:49,523 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1d9a242c]) (logid:d888b9c8) HA health check task is running... 2025-06-29 02:45:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f3efcfa]) (logid:40ed2c0a) No inactive management server node found 2025-06-29 02:45:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f3efcfa]) (logid:40ed2c0a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee3b8361]) (logid:c06228fc) No inactive management server node found 2025-06-29 02:45:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee3b8361]) (logid:c06228fc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-334183e7]) (logid:3bb03008) No inactive management server node found 2025-06-29 02:45:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-334183e7]) (logid:3bb03008) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:53,525 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0875f51f]) (logid:e6eb699a) HA health check task is running... 2025-06-29 02:45:54,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acf65a42]) (logid:52d74d22) No inactive management server node found 2025-06-29 02:45:54,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acf65a42]) (logid:52d74d22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:55,060 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7872: Seq 7-7872: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:45:55,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7872 2025-06-29 02:45:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6ae418d]) (logid:a616fc67) No inactive management server node found 2025-06-29 02:45:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6ae418d]) (logid:a616fc67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-de1bc3a8]) (logid:9a9edef5) Enabled non-edge zones available for scan: 1 2025-06-29 02:45:56,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-de1bc3a8]) (logid:9a9edef5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2315 TB], threshold=[90.00%]. 2025-06-29 02:45:56,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-de1bc3a8]) (logid:9a9edef5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:45:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2752f5dc]) (logid:5e4b2774) Enabled non-edge zones available for scan: 1 2025-06-29 02:45:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2752f5dc]) (logid:5e4b2774) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:45:57,527 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-33d1750a]) (logid:cba9e6a2) HA health check task is running... 2025-06-29 02:45:57,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0424238]) (logid:a64890cd) No inactive management server node found 2025-06-29 02:45:57,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0424238]) (logid:a64890cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:45:57,868 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-f0ea88a6]) (logid:f22b3a9a) AutoScaling Monitor is running... 2025-06-29 02:45:57,964 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-bad96e29]) (logid:63fbcdff) Starting to remove all volume_stats rows older than [Sat Jun 28 14:45:57 CEST 2025]. 2025-06-29 02:45:57,964 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-bad96e29]) (logid:63fbcdff) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:45:57 CEST 2025]. 2025-06-29 02:45:58,710 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:45:58,713 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:45:58,713 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) Seq 2-3048092522799696616: Routed from 90520738888109 2025-06-29 02:45:58,725 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) Seq 1-3048092522799696616: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:45:58,773 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:45:58,773 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) Seq 2-1612007191621929888: Routed from 90520738888109 2025-06-29 02:45:58,817 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-0db2f915]) (logid:acfa29c6) Seq 2-1612007191621929888: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:45:59,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fd9de9a]) (logid:8838c591) No inactive management server node found 2025-06-29 02:45:59,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fd9de9a]) (logid:8838c591) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:00,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5134cf5]) (logid:c1093772) No inactive management server node found 2025-06-29 02:46:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5134cf5]) (logid:c1093772) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:01,528 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-542fd0f7]) (logid:5d4ac762) HA health check task is running... 2025-06-29 02:46:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ae638c4]) (logid:f00218ea) No inactive management server node found 2025-06-29 02:46:02,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ae638c4]) (logid:f00218ea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-261f4b75]) (logid:bda8066a) No inactive management server node found 2025-06-29 02:46:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-261f4b75]) (logid:bda8066a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:05,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7873: Seq 7-7873: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:46:05,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7873 2025-06-29 02:46:05,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba379019]) (logid:7affab64) No inactive management server node found 2025-06-29 02:46:05,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba379019]) (logid:7affab64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:05,529 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-55989ea3]) (logid:0128f492) HA health check task is running... 2025-06-29 02:46:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-493be103]) (logid:1b0c598a) No inactive management server node found 2025-06-29 02:46:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-493be103]) (logid:1b0c598a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49d7a23a]) (logid:fd7ef747) No inactive management server node found 2025-06-29 02:46:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49d7a23a]) (logid:fd7ef747) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:46:09,530 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6e337a0c]) (logid:62f52173) HA health check task is running... 2025-06-29 02:46:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-196f46c3]) (logid:ae4bf7f5) No inactive management server node found 2025-06-29 02:46:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-196f46c3]) (logid:ae4bf7f5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d4bad23b]) (logid:b45662c6) Found 0 routers to update status. 2025-06-29 02:46:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d4bad23b]) (logid:b45662c6) Found 0 VPC's to update Redundant State. 2025-06-29 02:46:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d4bad23b]) (logid:b45662c6) Found 0 networks to update RvR status. 2025-06-29 02:46:11,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32d38962]) (logid:99fa18d0) No inactive management server node found 2025-06-29 02:46:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32d38962]) (logid:99fa18d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5225e848]) (logid:cf9f20ad) No inactive management server node found 2025-06-29 02:46:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5225e848]) (logid:cf9f20ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:12,911 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-5b93e376]) (logid:1687eeee) Starting to remove all vm_stats rows older than [Sat Jun 28 14:46:12 CEST 2025]. 2025-06-29 02:46:12,934 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-5b93e376]) (logid:1687eeee) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:46:12 CEST 2025]. 2025-06-29 02:46:13,532 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cef3e759]) (logid:9fc3f764) HA health check task is running... 2025-06-29 02:46:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c962dd77]) (logid:bd11caa5) No inactive management server node found 2025-06-29 02:46:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c962dd77]) (logid:bd11caa5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:15,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-7874: Seq 7-7874: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:46:15,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 7874 2025-06-29 02:46:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a57955af]) (logid:24e241c2) No inactive management server node found 2025-06-29 02:46:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a57955af]) (logid:24e241c2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:17,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d131f6f6]) (logid:091a7c37) Resetting hosts suitable for reconnect 2025-06-29 02:46:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d131f6f6]) (logid:091a7c37) Completed resetting hosts suitable for reconnect 2025-06-29 02:46:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d131f6f6]) (logid:091a7c37) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:46:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d131f6f6]) (logid:091a7c37) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:46:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d131f6f6]) (logid:091a7c37) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:46:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d131f6f6]) (logid:091a7c37) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:46:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6423069e]) (logid:814579bd) No inactive management server node found 2025-06-29 02:46:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6423069e]) (logid:814579bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:17,533 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-fac34c40]) (logid:f53687c2) HA health check task is running... 2025-06-29 02:46:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb7f71ad]) (logid:8c6379a6) No inactive management server node found 2025-06-29 02:46:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb7f71ad]) (logid:8c6379a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:20,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5357fb6]) (logid:458336b9) No inactive management server node found 2025-06-29 02:46:20,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5357fb6]) (logid:458336b9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:46:21,534 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d85ccd45]) (logid:74baa460) HA health check task is running... 2025-06-29 02:46:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cf108d3]) (logid:82f6fcda) No inactive management server node found 2025-06-29 02:46:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cf108d3]) (logid:82f6fcda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:23,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3ade012]) (logid:a55d482e) No inactive management server node found 2025-06-29 02:46:23,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3ade012]) (logid:a55d482e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d275900]) (logid:47a65ade) No inactive management server node found 2025-06-29 02:46:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d275900]) (logid:47a65ade) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:25,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7875: Seq 7-7875: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:46:25,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7875 2025-06-29 02:46:25,536 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bc6c6b06]) (logid:18b4cfd2) HA health check task is running... 2025-06-29 02:46:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-105a60ed]) (logid:f83b2cb4) No inactive management server node found 2025-06-29 02:46:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-105a60ed]) (logid:f83b2cb4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-da70e88a]) (logid:bc12fe06) Enabled non-edge zones available for scan: 1 2025-06-29 02:46:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-da70e88a]) (logid:bc12fe06) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2315 TB], threshold=[90.00%]. 2025-06-29 02:46:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-da70e88a]) (logid:bc12fe06) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:46:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0addea04]) (logid:0441b14f) Enabled non-edge zones available for scan: 1 2025-06-29 02:46:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0addea04]) (logid:0441b14f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:46:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bd1e950]) (logid:f67aa1ca) No inactive management server node found 2025-06-29 02:46:27,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bd1e950]) (logid:f67aa1ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:29,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed3a2d07]) (logid:a466dcfb) No inactive management server node found 2025-06-29 02:46:29,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed3a2d07]) (logid:a466dcfb) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:46:29,430 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:46:29,431 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:46:29,433 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) Seq 2-3048092522799696617: Routed from 90520738888109 2025-06-29 02:46:29,476 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) Seq 1-3048092522799696617: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:46:29,477 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:46:29,478 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) Seq 2-1612007191621929889: Routed from 90520738888109 2025-06-29 02:46:29,481 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8c02aece]) (logid:ada760ba) Seq 2-1612007191621929889: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:46:29,537 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a118a30e]) (logid:def0f518) HA health check task is running... 2025-06-29 02:46:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6a97e1b]) (logid:3b23420c) No inactive management server node found 2025-06-29 02:46:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6a97e1b]) (logid:3b23420c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:32,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f003edec]) (logid:d0bb4a1f) No inactive management server node found 2025-06-29 02:46:32,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f003edec]) (logid:d0bb4a1f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:33,538 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-962d9b65]) (logid:2446c3b1) HA health check task is running... 2025-06-29 02:46:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e14b84a]) (logid:4b559a8c) No inactive management server node found 2025-06-29 02:46:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e14b84a]) (logid:4b559a8c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:46:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:46:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:46:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:46:35,063 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7876: Seq 7-7876: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:46:35,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7876 2025-06-29 02:46:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69f8a71a]) (logid:1ed6f6a6) No inactive management server node found 2025-06-29 02:46:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69f8a71a]) (logid:1ed6f6a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42431b8a]) (logid:5a4a90f2) No inactive management server node found 2025-06-29 02:46:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42431b8a]) (logid:5a4a90f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:37,540 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3bafd3fd]) (logid:d59e242d) HA health check task is running... 2025-06-29 02:46:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a07eda99]) (logid:73871a3d) No inactive management server node found 2025-06-29 02:46:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a07eda99]) (logid:73871a3d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:39,724 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) StorageCollector is running... 2025-06-29 02:46:39,730 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:46:39,730 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:46:39,730 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:46:39,731 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-6717118844223095463: Routed from 90520738888109 2025-06-29 02:46:39,784 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 4-6717118844223095463: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:46:39,788 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:46:39,788 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:46:39,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:46:39,788 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-1612007191621929890: Routed from 90520738888109 2025-06-29 02:46:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-851315e3]) (logid:a8391eca) No inactive management server node found 2025-06-29 02:46:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-851315e3]) (logid:a8391eca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:39,840 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-1612007191621929890: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:46:39,850 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:46:39,850 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:46:39,850 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:46:39,851 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-1612007191621929891: Routed from 90520738888109 2025-06-29 02:46:39,943 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-1612007191621929891: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:46:39,954 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:46:39,955 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:46:39,955 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:46:39,956 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-3048092522799696618: Routed from 90520738888109 2025-06-29 02:46:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-86dc7974]) (logid:b1f73d82) Found 0 routers to update status. 2025-06-29 02:46:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-86dc7974]) (logid:b1f73d82) Found 0 VPC's to update Redundant State. 2025-06-29 02:46:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-86dc7974]) (logid:b1f73d82) Found 0 networks to update RvR status. 2025-06-29 02:46:41,007 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 1-3048092522799696618: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:46:41,009 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:46:41,009 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:46:41,009 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:46:41,010 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-1612007191621929892: Routed from 90520738888109 2025-06-29 02:46:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-b862690a]) (logid:f4fbdce5) DbCollector is running... 2025-06-29 02:46:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-046ef5e8]) (logid:3929d5fa) No inactive management server node found 2025-06-29 02:46:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-046ef5e8]) (logid:3929d5fa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:41,541 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-714e11db]) (logid:e5cd43c9) HA health check task is running... 2025-06-29 02:46:42,078 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-326f7fab]) (logid:49e565a6) Seq 2-1612007191621929892: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:46:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) ManagementServerCollector is running... 2025-06-29 02:46:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) DB properties were already loaded 2025-06-29 02:46:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:46:42,168 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Execution of process [4061994] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:46:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Exit value of process [4061994] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:46:42,168 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Process [4061994] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:46:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) usage install: null 2025-06-29 02:46:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Sessions active: 0 2025-06-29 02:46:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:46:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4061997] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:46:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:46:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062002] for command [/bin/bash -c uname -r ]. 2025-06-29 02:46:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:46:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062003] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:46:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) system memory from /proc: 4887515136 2025-06-29 02:46:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:46:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062008] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:46:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) free memory from /proc: 510070784 2025-06-29 02:46:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:46:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062013] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:46:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) used memory from /proc: 756108 2025-06-29 02:46:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:46:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062014] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:46:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:46:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062015] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:46:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:46:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062016] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:46:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:46:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062023] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:46:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:46:42,246 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Successfully executed process [4062024] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail buffersdirect.count, 120 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail buffersdirect.used, 657986 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail buffersmapped.count, 7 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail buffersmapped.used, 798602 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail gcPS-Scavenge.count, 479 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail gcPS-Scavenge.time, 3521 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail jvmuptime, 41942325 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memoryheap.committed, 350748672 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memoryheap.init, 77594624 2025-06-29 02:46:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memoryheap.usage, 0.10489738986113019 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorynon-heap.committed, 224854016 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorynon-heap.max, -1 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorynon-heap.usage, -2.20103472E8 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorynon-heap.used, 220103472 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17537719108237804 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.PS-Eden-Space.usage, 0.036954288323599795 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.PS-Old-Gen.usage, 0.11668159096549169 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorypools.PS-Survivor-Space.usage, 0.6345188306725543 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorytotal.committed, 575602688 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorytotal.init, 85262336 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorytotal.max, 1908932607 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail memorytotal.used, 420394824 2025-06-29 02:46:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail threadsdeadlocks, [] 2025-06-29 02:46:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail threadsnew.count, 0 2025-06-29 02:46:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:46:42,253 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-e7b581d7]) (logid:d500ef58) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:46:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":151251080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41942246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":419617480,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200242048,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510070784,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.26,0.23],"systemCyclesUsage":[66362106,53802870,4587505162],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:46:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1397, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:46:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":151251080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41942246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":419617480,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200242048,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510070784,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.26,0.23],"systemCyclesUsage":[66362106,53802870,4587505162],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:46:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:46:42,265 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 10 ms 2025-06-29 02:46:42,265 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 11ms. agent: 0, pdu seq: 1397, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:46:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":151251080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41942246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":419617480,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200242048,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510070784,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.26,0.23],"systemCyclesUsage":[66362106,53802870,4587505162],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:46:42,265 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-313:[ctx-e68b29d3]) (logid:a0d105b7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:46:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":151251080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":41942246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":419617480,"totalCommitted":575602688,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200242048,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510070784,"systemMemoryUsed":756108,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.26,0.23],"systemCyclesUsage":[66362106,53802870,4587505162],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:46:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cbb04d9]) (logid:2b137616) No inactive management server node found 2025-06-29 02:46:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cbb04d9]) (logid:2b137616) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c442e05c]) (logid:c261eac8) No inactive management server node found 2025-06-29 02:46:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c442e05c]) (logid:c261eac8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:45,062 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7877: Seq 7-7877: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:46:45,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7877 2025-06-29 02:46:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:46:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:46:45,542 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-97374b05]) (logid:a27ba653) HA health check task is running... 2025-06-29 02:46:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5dc333d1]) (logid:fd86296b) No inactive management server node found 2025-06-29 02:46:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5dc333d1]) (logid:fd86296b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f040f1a7]) (logid:6a7310f8) VM scheduler.poll is being called at 2025-06-29T00:47:00+0000 2025-06-29 02:46:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f040f1a7]) (logid:6a7310f8) Got 0 scheduled jobs to be executed at 2025-06-29T00:47:00+0000 2025-06-29 02:46:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f040f1a7]) (logid:6a7310f8) Cleaned up 0 VM scheduled job entries 2025-06-29 02:46:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-7eecb1f6]) (logid:ebb4a4a0) VM Operation Thread Running 2025-06-29 02:46:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-249d2ef7]) (logid:d0280efc) ClusterDRS.poll is being called at 2025-06-29T00:47:00+0000 2025-06-29 02:46:46,337 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-249d2ef7]) (logid:d0280efc) Removed 0 old drs migration plans 2025-06-29 02:46:47,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:46:47,229 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:46:47,231 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:46:47,233 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:46:47,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a75c1182]) (logid:c02a50ad) No inactive management server node found 2025-06-29 02:46:47,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a75c1182]) (logid:c02a50ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:48,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61e0886b]) (logid:82f844cc) No inactive management server node found 2025-06-29 02:46:48,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61e0886b]) (logid:82f844cc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:49,543 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-38927e32]) (logid:ffbe8c88) HA health check task is running... 2025-06-29 02:46:50,063 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-7879: Seq 7-7879: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:46:50,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 7879 2025-06-29 02:46:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a22c2565]) (logid:7dc5e68c) No inactive management server node found 2025-06-29 02:46:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a22c2565]) (logid:7dc5e68c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:51,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-659d55e9]) (logid:8ae8316f) No inactive management server node found 2025-06-29 02:46:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-659d55e9]) (logid:8ae8316f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-223d0611]) (logid:26df6da8) No inactive management server node found 2025-06-29 02:46:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-223d0611]) (logid:26df6da8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:53,544 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d7cd0aaa]) (logid:ee34e1b5) HA health check task is running... 2025-06-29 02:46:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eacc269c]) (logid:214d25ae) No inactive management server node found 2025-06-29 02:46:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eacc269c]) (logid:214d25ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:56,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44e9dca8]) (logid:0ab48735) No inactive management server node found 2025-06-29 02:46:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44e9dca8]) (logid:0ab48735) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-08482149]) (logid:732769a6) Enabled non-edge zones available for scan: 1 2025-06-29 02:46:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-08482149]) (logid:732769a6) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2319 TB], threshold=[90.00%]. 2025-06-29 02:46:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-08482149]) (logid:732769a6) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:46:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b6e70895]) (logid:1e6f4807) Enabled non-edge zones available for scan: 1 2025-06-29 02:46:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b6e70895]) (logid:1e6f4807) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:46:57,546 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e6718299]) (logid:82f8f021) HA health check task is running... 2025-06-29 02:46:57,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-260263ec]) (logid:a97a3ea4) No inactive management server node found 2025-06-29 02:46:57,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-260263ec]) (logid:a97a3ea4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:46:57,870 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-181e1211]) (logid:4d5ef79a) AutoScaling Monitor is running... 2025-06-29 02:46:57,966 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-eee9aa68]) (logid:1f1d79bd) Starting to remove all volume_stats rows older than [Sat Jun 28 14:46:57 CEST 2025]. 2025-06-29 02:46:57,966 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-eee9aa68]) (logid:1f1d79bd) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:46:57 CEST 2025]. 2025-06-29 02:46:58,873 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:46:58,876 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:46:58,877 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) Seq 2-3048092522799696619: Routed from 90520738888109 2025-06-29 02:46:58,888 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) Seq 1-3048092522799696619: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:46:58,936 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:46:58,936 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) Seq 2-1612007191621929893: Routed from 90520738888109 2025-06-29 02:46:58,982 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4651194b]) (logid:57bf123d) Seq 2-1612007191621929893: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:46:59,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81a824bb]) (logid:2a851f6e) No inactive management server node found 2025-06-29 02:46:59,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81a824bb]) (logid:2a851f6e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:00,063 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7880: Seq 7-7880: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:47:00,072 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7880 2025-06-29 02:47:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7352d95]) (logid:5eec9932) No inactive management server node found 2025-06-29 02:47:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7352d95]) (logid:5eec9932) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:01,547 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c20776f6]) (logid:66e8ad72) HA health check task is running... 2025-06-29 02:47:02,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7761ad55]) (logid:ad09657a) No inactive management server node found 2025-06-29 02:47:02,338 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7761ad55]) (logid:ad09657a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9c43aef]) (logid:22a3e08b) No inactive management server node found 2025-06-29 02:47:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9c43aef]) (logid:22a3e08b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e3081a0]) (logid:2679dab4) No inactive management server node found 2025-06-29 02:47:05,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e3081a0]) (logid:2679dab4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:05,548 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2145859f]) (logid:3dbb4503) HA health check task is running... 2025-06-29 02:47:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb4c027a]) (logid:e9b50a04) No inactive management server node found 2025-06-29 02:47:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb4c027a]) (logid:e9b50a04) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:08,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9896d0e]) (logid:79461061) No inactive management server node found 2025-06-29 02:47:08,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9896d0e]) (logid:79461061) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:09,550 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5e7cf30b]) (logid:e951c2a7) HA health check task is running... 2025-06-29 02:47:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54cbee24]) (logid:55538e58) No inactive management server node found 2025-06-29 02:47:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54cbee24]) (logid:55538e58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:10,064 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7881: Seq 7-7881: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:47:10,074 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7881 2025-06-29 02:47:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ef8c311c]) (logid:04e1c1f0) Found 0 routers to update status. 2025-06-29 02:47:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ef8c311c]) (logid:04e1c1f0) Found 0 VPC's to update Redundant State. 2025-06-29 02:47:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ef8c311c]) (logid:04e1c1f0) Found 0 networks to update RvR status. 2025-06-29 02:47:11,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6918795]) (logid:1d833fb1) No inactive management server node found 2025-06-29 02:47:11,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6918795]) (logid:1d833fb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:12,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8275d553]) (logid:bd1c406f) No inactive management server node found 2025-06-29 02:47:12,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8275d553]) (logid:bd1c406f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:12,936 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-539e606e]) (logid:dd3d2124) Starting to remove all vm_stats rows older than [Sat Jun 28 14:47:12 CEST 2025]. 2025-06-29 02:47:12,960 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-539e606e]) (logid:dd3d2124) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:47:12 CEST 2025]. 2025-06-29 02:47:13,551 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f92ff912]) (logid:46ca9404) HA health check task is running... 2025-06-29 02:47:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94b92827]) (logid:5071c791) No inactive management server node found 2025-06-29 02:47:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94b92827]) (logid:5071c791) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69a7a999]) (logid:58bf0163) No inactive management server node found 2025-06-29 02:47:15,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69a7a999]) (logid:58bf0163) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:17,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a15c7ed]) (logid:09283bda) No inactive management server node found 2025-06-29 02:47:17,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a15c7ed]) (logid:09283bda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:17,552 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-89375909]) (logid:c852e751) HA health check task is running... 2025-06-29 02:47:18,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4146db98]) (logid:f2e98d09) No inactive management server node found 2025-06-29 02:47:18,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4146db98]) (logid:f2e98d09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:20,064 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7882: Seq 7-7882: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:47:20,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7882 2025-06-29 02:47:20,338 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f712cac]) (logid:7d94a741) No inactive management server node found 2025-06-29 02:47:20,338 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f712cac]) (logid:7d94a741) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:47:21,554 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7d5573ea]) (logid:0c1f4d6b) HA health check task is running... 2025-06-29 02:47:21,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c340fd5]) (logid:b824437c) No inactive management server node found 2025-06-29 02:47:21,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c340fd5]) (logid:b824437c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:47:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c26ef4e]) (logid:e320f3a5) No inactive management server node found 2025-06-29 02:47:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c26ef4e]) (logid:e320f3a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c59a0b24]) (logid:242c68c8) No inactive management server node found 2025-06-29 02:47:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c59a0b24]) (logid:242c68c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:25,555 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1355edae]) (logid:032ba62c) HA health check task is running... 2025-06-29 02:47:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac849b93]) (logid:ca7a0143) No inactive management server node found 2025-06-29 02:47:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac849b93]) (logid:ca7a0143) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e7a4d98f]) (logid:0d36b144) Enabled non-edge zones available for scan: 1 2025-06-29 02:47:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e7a4d98f]) (logid:0d36b144) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2319 TB], threshold=[90.00%]. 2025-06-29 02:47:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e7a4d98f]) (logid:0d36b144) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:47:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-445685b6]) (logid:17755e03) Enabled non-edge zones available for scan: 1 2025-06-29 02:47:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-445685b6]) (logid:17755e03) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:47:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1aa9465]) (logid:ba9030d5) No inactive management server node found 2025-06-29 02:47:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1aa9465]) (logid:ba9030d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:29,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb101d5b]) (logid:025aba1c) No inactive management server node found 2025-06-29 02:47:29,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb101d5b]) (logid:025aba1c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:29,484 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:47:29,486 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:47:29,487 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) Seq 2-3048092522799696620: Routed from 90520738888109 2025-06-29 02:47:29,530 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) Seq 1-3048092522799696620: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:47:29,531 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:47:29,532 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) Seq 2-1612007191621929894: Routed from 90520738888109 2025-06-29 02:47:29,535 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-70e211de]) (logid:3f01a299) Seq 2-1612007191621929894: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:47:29,556 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-372d230b]) (logid:0d4ec11a) HA health check task is running... 2025-06-29 02:47:30,065 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7883: Seq 7-7883: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:47:30,074 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7883 2025-06-29 02:47:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c08b6a1]) (logid:161030c9) No inactive management server node found 2025-06-29 02:47:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c08b6a1]) (logid:161030c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a27f969f]) (logid:0243e2c3) No inactive management server node found 2025-06-29 02:47:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a27f969f]) (logid:0243e2c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:33,558 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-12a5f6c9]) (logid:5a336bdd) HA health check task is running... 2025-06-29 02:47:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4633e4ee]) (logid:375ecd88) No inactive management server node found 2025-06-29 02:47:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4633e4ee]) (logid:375ecd88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:47:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:47:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:47:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:47:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02ce9009]) (logid:c8442b87) No inactive management server node found 2025-06-29 02:47:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02ce9009]) (logid:c8442b87) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d4cb1f2]) (logid:3f99907d) No inactive management server node found 2025-06-29 02:47:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d4cb1f2]) (logid:3f99907d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:37,559 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-47c2fe07]) (logid:dd13bba8) HA health check task is running... 2025-06-29 02:47:38,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-808a21b3]) (logid:f51cf142) No inactive management server node found 2025-06-29 02:47:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-808a21b3]) (logid:f51cf142) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:47:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cc6e4f2]) (logid:cad84369) No inactive management server node found 2025-06-29 02:47:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cc6e4f2]) (logid:cad84369) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:40,066 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7884: Seq 7-7884: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:47:40,075 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7884 2025-06-29 02:47:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c5e1ffdf]) (logid:64217988) Found 0 routers to update status. 2025-06-29 02:47:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c5e1ffdf]) (logid:64217988) Found 0 VPC's to update Redundant State. 2025-06-29 02:47:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c5e1ffdf]) (logid:64217988) Found 0 networks to update RvR status. 2025-06-29 02:47:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-e29fc417]) (logid:8aa4c7a9) DbCollector is running... 2025-06-29 02:47:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74ad1f50]) (logid:60f78bc9) No inactive management server node found 2025-06-29 02:47:41,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74ad1f50]) (logid:60f78bc9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:41,560 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b9c7265b]) (logid:416ddd6f) HA health check task is running... 2025-06-29 02:47:42,078 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) StorageCollector is running... 2025-06-29 02:47:42,084 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:47:42,084 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:47:42,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:47:42,085 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 2-6717118844223095464: Routed from 90520738888109 2025-06-29 02:47:42,140 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 4-6717118844223095464: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:47:42,144 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:47:42,144 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:47:42,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:47:42,146 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 2-3048092522799696621: Routed from 90520738888109 2025-06-29 02:47:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) ManagementServerCollector is running... 2025-06-29 02:47:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) DB properties were already loaded 2025-06-29 02:47:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:47:42,167 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Execution of process [4062030] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:47:42,167 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Exit value of process [4062030] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:47:42,167 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Process [4062030] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:47:42,167 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) usage install: null 2025-06-29 02:47:42,167 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Sessions active: 0 2025-06-29 02:47:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:47:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062033] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:47:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:47:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062038] for command [/bin/bash -c uname -r ]. 2025-06-29 02:47:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:47:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062039] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:47:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) system memory from /proc: 4887515136 2025-06-29 02:47:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:47:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062044] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:47:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) free memory from /proc: 507879424 2025-06-29 02:47:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:47:42,194 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 1-3048092522799696621: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:47:42,205 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:47:42,205 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:47:42,205 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:47:42,207 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 2-3048092522799696622: Routed from 90520738888109 2025-06-29 02:47:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062049] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:47:42,210 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) used memory from /proc: 755956 2025-06-29 02:47:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:47:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062050] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:47:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:47:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062051] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:47:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:47:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062052] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:47:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:47:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062059] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:47:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:47:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Successfully executed process [4062060] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail buffersdirect.count, 120 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail buffersdirect.used, 657986 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail buffersmapped.count, 7 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail buffersmapped.used, 798602 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail gcPS-Scavenge.count, 480 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail gcPS-Scavenge.time, 3527 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail jvmuptime, 42002329 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memoryheap.committed, 350748672 2025-06-29 02:47:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memoryheap.init, 77594624 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memoryheap.usage, 0.103326614660668 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorynon-heap.max, -1 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorynon-heap.usage, -2.20124848E8 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorynon-heap.used, 220124848 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17555110303918955 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.PS-Eden-Space.usage, 0.03263116316361861 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.PS-Old-Gen.usage, 0.11715646280730788 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorypools.PS-Survivor-Space.usage, 0.576091600501019 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorytotal.committed, 575668224 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorytotal.init, 85262336 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorytotal.max, 1908932607 2025-06-29 02:47:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail memorytotal.used, 417419824 2025-06-29 02:47:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail threadsdeadlocks, [] 2025-06-29 02:47:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail threadsnew.count, 0 2025-06-29 02:47:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) not storing detail threadstimed_waiting.count, 86 2025-06-29 02:47:42,255 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-6c9a85a4]) (logid:1ffb155d) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:47:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":154480056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42002245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.38,"totalInit":85262336,"totalUsed":416393464,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197243544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":507879424,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.38,0.31,0.26],"systemCyclesUsage":[66362417,53803179,4587528298],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:47:42,255 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1399, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:47:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":154480056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42002245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.38,"totalInit":85262336,"totalUsed":416393464,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197243544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":507879424,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.38,0.31,0.26],"systemCyclesUsage":[66362417,53803179,4587528298],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:47:42,256 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:47:42,260 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-313:[ctx-553476c2]) (logid:73e72cb7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:47:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":154480056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42002245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.38,"totalInit":85262336,"totalUsed":416393464,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197243544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":507879424,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.38,0.31,0.26],"systemCyclesUsage":[66362417,53803179,4587528298],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:47:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:47:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1399, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:47:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":154480056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42002245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.38,"totalInit":85262336,"totalUsed":416393464,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197243544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":507879424,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.38,0.31,0.26],"systemCyclesUsage":[66362417,53803179,4587528298],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:47:42,295 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 1-3048092522799696622: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:47:42,309 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:47:42,309 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:47:42,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:47:42,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 2-3048092522799696623: Routed from 90520738888109 2025-06-29 02:47:42,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a43295d5]) (logid:76230d96) No inactive management server node found 2025-06-29 02:47:42,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a43295d5]) (logid:76230d96) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:43,403 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 1-3048092522799696623: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:47:43,406 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:47:43,406 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:47:43,406 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:47:43,407 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 2-1612007191621929895: Routed from 90520738888109 2025-06-29 02:47:44,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13d61ea0]) (logid:bb9fc300) No inactive management server node found 2025-06-29 02:47:44,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13d61ea0]) (logid:bb9fc300) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:44,466 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-80ebf9bc]) (logid:3a5fa992) Seq 2-1612007191621929895: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:47:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:47:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:47:45,562 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-780d47c9]) (logid:c6b41285) HA health check task is running... 2025-06-29 02:47:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b11db8b3]) (logid:f3950bd1) No inactive management server node found 2025-06-29 02:47:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b11db8b3]) (logid:f3950bd1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a1ab2de1]) (logid:be12e5e5) VM scheduler.poll is being called at 2025-06-29T00:48:00+0000 2025-06-29 02:47:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a1ab2de1]) (logid:be12e5e5) Got 0 scheduled jobs to be executed at 2025-06-29T00:48:00+0000 2025-06-29 02:47:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a1ab2de1]) (logid:be12e5e5) Cleaned up 0 VM scheduled job entries 2025-06-29 02:47:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-70eefdd7]) (logid:86890e81) VM Operation Thread Running 2025-06-29 02:47:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-14871cab]) (logid:6bed173d) ClusterDRS.poll is being called at 2025-06-29T00:48:00+0000 2025-06-29 02:47:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-14871cab]) (logid:6bed173d) Removed 0 old drs migration plans 2025-06-29 02:47:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c72d068b]) (logid:4b41e593) Resetting hosts suitable for reconnect 2025-06-29 02:47:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c72d068b]) (logid:4b41e593) Completed resetting hosts suitable for reconnect 2025-06-29 02:47:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c72d068b]) (logid:4b41e593) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:47:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c72d068b]) (logid:4b41e593) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:47:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c72d068b]) (logid:4b41e593) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:47:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c72d068b]) (logid:4b41e593) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:47:47,222 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:47:47,224 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:47:47,226 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:47:47,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:47:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ec60d75]) (logid:cb086ad3) No inactive management server node found 2025-06-29 02:47:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ec60d75]) (logid:cb086ad3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:48,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a578381b]) (logid:377709ed) No inactive management server node found 2025-06-29 02:47:48,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a578381b]) (logid:377709ed) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:49,563 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-20599077]) (logid:3445893f) HA health check task is running... 2025-06-29 02:47:50,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7886: Seq 7-7886: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:47:50,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7886 2025-06-29 02:47:50,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aea4ca6b]) (logid:46d5e3bb) No inactive management server node found 2025-06-29 02:47:50,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aea4ca6b]) (logid:46d5e3bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:51,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a0ddf80]) (logid:b9c780ca) No inactive management server node found 2025-06-29 02:47:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a0ddf80]) (logid:b9c780ca) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-054a545e]) (logid:fa580204) No inactive management server node found 2025-06-29 02:47:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-054a545e]) (logid:fa580204) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:53,564 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3006fb91]) (logid:148922d6) HA health check task is running... 2025-06-29 02:47:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f337343]) (logid:f550c199) No inactive management server node found 2025-06-29 02:47:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f337343]) (logid:f550c199) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03a07290]) (logid:fd08ddd4) No inactive management server node found 2025-06-29 02:47:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03a07290]) (logid:fd08ddd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-64b9b9b2]) (logid:25f6d14a) Enabled non-edge zones available for scan: 1 2025-06-29 02:47:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-64b9b9b2]) (logid:25f6d14a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2323 TB], threshold=[90.00%]. 2025-06-29 02:47:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-64b9b9b2]) (logid:25f6d14a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:47:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dc20c1af]) (logid:a89b06b7) Enabled non-edge zones available for scan: 1 2025-06-29 02:47:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dc20c1af]) (logid:a89b06b7) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:47:57,565 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8bc50125]) (logid:3709fbd2) HA health check task is running... 2025-06-29 02:47:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee08202f]) (logid:48dc6af3) No inactive management server node found 2025-06-29 02:47:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee08202f]) (logid:48dc6af3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:47:57,871 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-37920217]) (logid:ba963aa7) AutoScaling Monitor is running... 2025-06-29 02:47:57,969 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-13dc215a]) (logid:6a7ab214) Starting to remove all volume_stats rows older than [Sat Jun 28 14:47:57 CEST 2025]. 2025-06-29 02:47:57,969 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-13dc215a]) (logid:6a7ab214) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:47:57 CEST 2025]. 2025-06-29 02:47:59,040 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:47:59,043 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:47:59,044 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) Seq 2-3048092522799696624: Routed from 90520738888109 2025-06-29 02:47:59,057 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) Seq 1-3048092522799696624: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:47:59,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:47:59,092 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) Seq 2-1612007191621929896: Routed from 90520738888109 2025-06-29 02:47:59,141 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-12ff8382]) (logid:4e2fa987) Seq 2-1612007191621929896: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:47:59,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07d2b647]) (logid:216e4394) No inactive management server node found 2025-06-29 02:47:59,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07d2b647]) (logid:216e4394) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:48:00,066 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7887: Seq 7-7887: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:00,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7887 2025-06-29 02:48:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e75314d]) (logid:975d73f5) No inactive management server node found 2025-06-29 02:48:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e75314d]) (logid:975d73f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:01,567 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c9082c71]) (logid:245f7e88) HA health check task is running... 2025-06-29 02:48:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9af44784]) (logid:62265ba3) No inactive management server node found 2025-06-29 02:48:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9af44784]) (logid:62265ba3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c58b19]) (logid:025676e1) No inactive management server node found 2025-06-29 02:48:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c58b19]) (logid:025676e1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:05,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1229ed55]) (logid:e9ce1c38) No inactive management server node found 2025-06-29 02:48:05,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1229ed55]) (logid:e9ce1c38) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:05,568 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e908bac7]) (logid:03dc16c7) HA health check task is running... 2025-06-29 02:48:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe10609c]) (logid:54d5488b) No inactive management server node found 2025-06-29 02:48:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe10609c]) (logid:54d5488b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:08,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45b2c7c9]) (logid:8c622a57) No inactive management server node found 2025-06-29 02:48:08,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45b2c7c9]) (logid:8c622a57) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:09,569 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-25251b80]) (logid:72865e48) HA health check task is running... 2025-06-29 02:48:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a9f7316]) (logid:7c64249e) No inactive management server node found 2025-06-29 02:48:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a9f7316]) (logid:7c64249e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:10,067 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7888: Seq 7-7888: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:10,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7888 2025-06-29 02:48:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5001d407]) (logid:8931228d) Found 0 routers to update status. 2025-06-29 02:48:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5001d407]) (logid:8931228d) Found 0 VPC's to update Redundant State. 2025-06-29 02:48:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5001d407]) (logid:8931228d) Found 0 networks to update RvR status. 2025-06-29 02:48:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c29f3cc]) (logid:40130707) No inactive management server node found 2025-06-29 02:48:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c29f3cc]) (logid:40130707) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:48:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ed689df]) (logid:7c32b932) No inactive management server node found 2025-06-29 02:48:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ed689df]) (logid:7c32b932) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:12,962 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-23dc4578]) (logid:9710f558) Starting to remove all vm_stats rows older than [Sat Jun 28 14:48:12 CEST 2025]. 2025-06-29 02:48:12,985 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-23dc4578]) (logid:9710f558) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:48:12 CEST 2025]. 2025-06-29 02:48:13,571 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-cc626598]) (logid:2efa2e5a) HA health check task is running... 2025-06-29 02:48:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfc77872]) (logid:28d934cf) No inactive management server node found 2025-06-29 02:48:14,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfc77872]) (logid:28d934cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7dc63aa]) (logid:370a8105) No inactive management server node found 2025-06-29 02:48:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7dc63aa]) (logid:370a8105) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1220563]) (logid:bf32edc4) No inactive management server node found 2025-06-29 02:48:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1220563]) (logid:bf32edc4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:17,572 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-90fa4d06]) (logid:db2d9b2d) HA health check task is running... 2025-06-29 02:48:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-102f441c]) (logid:90657aed) No inactive management server node found 2025-06-29 02:48:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-102f441c]) (logid:90657aed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:20,067 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7889: Seq 7-7889: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:20,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7889 2025-06-29 02:48:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e32fe75f]) (logid:9202bdd1) No inactive management server node found 2025-06-29 02:48:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e32fe75f]) (logid:9202bdd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:21,573 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7faaad90]) (logid:08a24b9b) HA health check task is running... 2025-06-29 02:48:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-401c2089]) (logid:d85c433a) No inactive management server node found 2025-06-29 02:48:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-401c2089]) (logid:d85c433a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-622c0219]) (logid:959350c8) No inactive management server node found 2025-06-29 02:48:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-622c0219]) (logid:959350c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:48:24,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ea55677]) (logid:0abcdd24) No inactive management server node found 2025-06-29 02:48:24,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ea55677]) (logid:0abcdd24) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:25,574 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d3ad5e01]) (logid:9860ea87) HA health check task is running... 2025-06-29 02:48:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5058012e]) (logid:3cbcd25d) No inactive management server node found 2025-06-29 02:48:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5058012e]) (logid:3cbcd25d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-858a98ff]) (logid:a485e1c0) Enabled non-edge zones available for scan: 1 2025-06-29 02:48:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-858a98ff]) (logid:a485e1c0) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2323 TB], threshold=[90.00%]. 2025-06-29 02:48:26,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-858a98ff]) (logid:a485e1c0) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:48:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b5f72c70]) (logid:d1fcc16b) Enabled non-edge zones available for scan: 1 2025-06-29 02:48:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b5f72c70]) (logid:d1fcc16b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:48:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-916223e8]) (logid:e456b4b0) No inactive management server node found 2025-06-29 02:48:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-916223e8]) (logid:e456b4b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-844bb7c2]) (logid:feb292d8) No inactive management server node found 2025-06-29 02:48:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-844bb7c2]) (logid:feb292d8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:29,538 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:48:29,539 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:48:29,540 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) Seq 2-3048092522799696625: Routed from 90520738888109 2025-06-29 02:48:29,576 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f21898fc]) (logid:0b6e8670) HA health check task is running... 2025-06-29 02:48:29,583 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) Seq 1-3048092522799696625: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:48:29,584 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:48:29,585 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) Seq 2-1612007191621929897: Routed from 90520738888109 2025-06-29 02:48:29,588 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e9e8e32b]) (logid:c6aedc39) Seq 2-1612007191621929897: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:48:30,067 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7890: Seq 7-7890: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:30,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7890 2025-06-29 02:48:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddef644f]) (logid:779d2789) No inactive management server node found 2025-06-29 02:48:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddef644f]) (logid:779d2789) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:32,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db941c73]) (logid:87de1798) No inactive management server node found 2025-06-29 02:48:32,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db941c73]) (logid:87de1798) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:33,577 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e39af90a]) (logid:b2132dfc) HA health check task is running... 2025-06-29 02:48:33,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9108bb58]) (logid:bbd2cb74) No inactive management server node found 2025-06-29 02:48:33,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9108bb58]) (logid:bbd2cb74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:48:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:48:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:48:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:48:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d15eaab8]) (logid:7e2360b6) No inactive management server node found 2025-06-29 02:48:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d15eaab8]) (logid:7e2360b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51bf353f]) (logid:53fd7df3) No inactive management server node found 2025-06-29 02:48:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51bf353f]) (logid:53fd7df3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:37,578 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-954a6371]) (logid:bf2998ed) HA health check task is running... 2025-06-29 02:48:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1244e8d8]) (logid:71813ffe) No inactive management server node found 2025-06-29 02:48:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1244e8d8]) (logid:71813ffe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ea6ac01]) (logid:9aba9a93) No inactive management server node found 2025-06-29 02:48:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ea6ac01]) (logid:9aba9a93) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:40,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7891: Seq 7-7891: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:40,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7891 2025-06-29 02:48:40,815 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-0579e27f]) (logid:fda00f13) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 02:48:40,816 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-0579e27f]) (logid:fda00f13) Snapshot scheduler is being called at [2025-06-29T00:48:40+0000]. 2025-06-29 02:48:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-0579e27f]) (logid:fda00f13) There are [0] scheduled snapshots to be executed at [2025-06-29T00:48:40+0000]. 2025-06-29 02:48:40,916 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-06fd283d]) (logid:20a1c51e) External devices are not used. Skipping external device usage collection 2025-06-29 02:48:40,946 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-d6056ed2]) (logid:2be0c9aa) checking health of usage server 2025-06-29 02:48:40,947 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-d6056ed2]) (logid:2be0c9aa) usage server running? false, heartbeat: null 2025-06-29 02:48:40,950 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-d6056ed2]) (logid:2be0c9aa) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 02:48:40,951 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-d6056ed2]) (logid:2be0c9aa) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 02:48:40,979 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-92f41085]) (logid:832a6353) Backup backup.poll is being called at 2025-06-29T00:48:40+0000 2025-06-29 02:48:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-ec6d0d1c]) (logid:b0eb9fe7) Found 0 running routers. 2025-06-29 02:48:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7f79380e]) (logid:7f4676ec) Found 0 routers to update status. 2025-06-29 02:48:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7f79380e]) (logid:7f4676ec) Found 0 VPC's to update Redundant State. 2025-06-29 02:48:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7f79380e]) (logid:7f4676ec) Found 0 networks to update RvR status. 2025-06-29 02:48:41,019 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Found 1 running routers. 2025-06-29 02:48:41,023 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 02:48:41,024 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Seq 2-1612007191621929898: Routed from 90520738888109 2025-06-29 02:48:41,024 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Seq 2-1612007191621929898: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:41,040 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-8516bb72]) (logid:0551a54c) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 02:48:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-0a27c072]) (logid:25ee1dc3) DbCollector is running... 2025-06-29 02:48:41,277 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-000dcbef]) (logid:ca8ab55e) Started resource counters recalculation periodic task. 2025-06-29 02:48:41,280 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-000dcbef]) (logid:ca8ab55e) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 02:48:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb673fa5]) (logid:ced229cd) No inactive management server node found 2025-06-29 02:48:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb673fa5]) (logid:ced229cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:41,579 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d7dbd0aa]) (logid:bfffaf46) HA health check task is running... 2025-06-29 02:48:41,581 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:[]) (logid:) Seq 2-1612007191621929898: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:41,581 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Seq 2-1612007191621929898: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 02:48:41,581 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 02:48:41,583 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 02:48:41,605 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-000dcbef]) (logid:ca8ab55e) Finished resource counters recalculation periodic task. 2025-06-29 02:48:41,625 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 02:48:41,627 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Seq 2-1612007191621929899: Routed from 90520738888109 2025-06-29 02:48:41,628 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Seq 2-1612007191621929899: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) ManagementServerCollector is running... 2025-06-29 02:48:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) DB properties were already loaded 2025-06-29 02:48:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:48:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Execution of process [4062073] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:48:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Exit value of process [4062073] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:48:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Process [4062073] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:48:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) usage install: null 2025-06-29 02:48:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Sessions active: 0 2025-06-29 02:48:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:48:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062076] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:48:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:48:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062081] for command [/bin/bash -c uname -r ]. 2025-06-29 02:48:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:48:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062082] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:48:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) system memory from /proc: 4887515136 2025-06-29 02:48:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:48:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062087] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:48:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) free memory from /proc: 517406720 2025-06-29 02:48:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:48:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062092] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:48:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) used memory from /proc: 755956 2025-06-29 02:48:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:48:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062093] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:48:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:48:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062094] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:48:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:48:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062095] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:48:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:48:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062102] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:48:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:48:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Successfully executed process [4062103] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail buffersdirect.count, 120 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail buffersdirect.used, 657986 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail buffersmapped.count, 7 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail buffersmapped.used, 798602 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail gcPS-Scavenge.count, 482 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail gcPS-Scavenge.time, 3540 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail jvmuptime, 42062322 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memoryheap.committed, 350748672 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memoryheap.init, 77594624 2025-06-29 02:48:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memoryheap.usage, 0.09511635939323847 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorynon-heap.max, -1 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorynon-heap.usage, -2.20132016E8 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorynon-heap.used, 220132016 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17560942082111436 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.PS-Eden-Space.usage, 0.009681083684974688 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.PS-Old-Gen.usage, 0.11829563114329687 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorypools.PS-Survivor-Space.usage, 0.4786966497247869 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorytotal.committed, 575668224 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorytotal.init, 85262336 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorytotal.max, 1908932607 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail memorytotal.used, 401761048 2025-06-29 02:48:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail threadsdeadlocks, [] 2025-06-29 02:48:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail threadsnew.count, 0 2025-06-29 02:48:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:48:42,249 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-4dc21b6e]) (logid:49220d25) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:48:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":169936952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42062247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":400943736,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":181570720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517406720,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.25,0.24],"systemCyclesUsage":[66362780,53803516,4587551330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:48:42,250 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1401, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:48:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":169936952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42062247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":400943736,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":181570720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517406720,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.25,0.24],"systemCyclesUsage":[66362780,53803516,4587551330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:48:42,250 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:48:42,253 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-313:[ctx-011aa8ce]) (logid:97b431d2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:48:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":169936952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42062247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":400943736,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":181570720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517406720,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.25,0.24],"systemCyclesUsage":[66362780,53803516,4587551330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:48:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:48:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1401, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:48:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":169936952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42062247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":400943736,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":181570720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":517406720,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.25,0.24],"systemCyclesUsage":[66362780,53803516,4587551330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:48:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e94fce9d]) (logid:7916084a) No inactive management server node found 2025-06-29 02:48:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e94fce9d]) (logid:7916084a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:43,029 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 2-1612007191621929899: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.e1aef073-0e27-470c-abb7-e355d207a35f","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:43,029 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6da348c5]) (logid:92b85ab3) Seq 2-1612007191621929899: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 02:48:43,031 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 02:48:43,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 02:48:43,036 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Seq 2-1612007191621929900: Routed from 90520738888109 2025-06-29 02:48:43,036 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Seq 2-1612007191621929900: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:43,474 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-1612007191621929900: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "14.412164688110352", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "27.24456787109375", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "40.70901870727539", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "54.00395393371582", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158082261", "lastRunDuration": "45.96877098083496", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158082337", "lastRunDuration": "8101.41921043396", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158090494", "lastRunDuration": "71.84505462646484", "message": "Memory Usage within limits with current at 50.8774%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158090566", "lastRunDuration": "3234.6138954162598", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158093844", "lastRunDuration": "42.53530502319336", "message": "Sufficient free space is 2571.59765625 MB"}, "lastRun": {"start": "2025-06-29 00:48:02.078967", "end": "2025-06-29 00:48:13.886838", "duration": "11.807870864868164"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751157601828", "lastRunDuration": "46.48184776306152", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751157601875", "lastRunDuration": "54.00204658508301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751157602056", "lastRunDuration": "55.10592460632324", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751157602219", "lastRunDuration": "45.26185989379883", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:40:01.825550", "end": "2025-06-29 00:40:02.303104", "duration": "0.4775538444519043"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "14.412164688110352", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "27.24456787109375", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "40.70901870727539", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "54.00395393371582", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158082261", "lastRunDuration": "45.96877098083496", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158082337", "lastRunDuration": "8101.41921043396", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158090494", "lastRunDuration": "71.84505462646484", "message": "Memory Usage within limits with current at 50.8774%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158090566", "lastRunDuration": "3234.6138954162598", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158093844", "lastRunDuration": "42.53530502319336", "message": "Sufficient free space is 2571.59765625 MB"}, "lastRun": {"start": "2025-06-29 00:48:02.078967", "end": "2025-06-29 00:48:13.886838", "duration": "11.807870864868164"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751157601828", "lastRunDuration": "46.48184776306152", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751157601875", "lastRunDuration": "54.00204658508301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751157602056", "lastRunDuration": "55.10592460632324", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751157602219", "lastRunDuration": "45.26185989379883", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:40:01.825550", "end": "2025-06-29 00:40:02.303104", "duration": "0.4775538444519043"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:43,474 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Seq 2-1612007191621929900: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 02:48:43,474 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "14.412164688110352", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "27.24456787109375", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "40.70901870727539", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "54.00395393371582", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158082261", "lastRunDuration": "45.96877098083496", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158082337", "lastRunDuration": "8101.41921043396", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158090494", "lastRunDuration": "71.84505462646484", "message": "Memory Usage within limits with current at 50.8774%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158090566", "lastRunDuration": "3234.6138954162598", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158093844", "lastRunDuration": "42.53530502319336", "message": "Sufficient free space is 2571.59765625 MB"}, "lastRun": {"start": "2025-06-29 00:48:02.078967", "end": "2025-06-29 00:48:13.886838", "duration": "11.807870864868164"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751157601828", "lastRunDuration": "46.48184776306152", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751157601875", "lastRunDuration": "54.00204658508301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751157602056", "lastRunDuration": "55.10592460632324", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751157602219", "lastRunDuration": "45.26185989379883", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:40:01.825550", "end": "2025-06-29 00:40:02.303104", "duration": "0.4775538444519043"}}} 2025-06-29 02:48:43,529 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "14.412164688110352", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "27.24456787109375", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "40.70901870727539", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158082082", "lastRunDuration": "54.00395393371582", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158082261", "lastRunDuration": "45.96877098083496", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158082337", "lastRunDuration": "8101.41921043396", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158090494", "lastRunDuration": "71.84505462646484", "message": "Memory Usage within limits with current at 50.8774%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158090566", "lastRunDuration": "3234.6138954162598", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158093844", "lastRunDuration": "42.53530502319336", "message": "Sufficient free space is 2571.59765625 MB"}, "lastRun": {"start": "2025-06-29 00:48:02.078967", "end": "2025-06-29 00:48:13.886838", "duration": "11.807870864868164"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751157601828", "lastRunDuration": "46.48184776306152", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751157601875", "lastRunDuration": "54.00204658508301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751157602056", "lastRunDuration": "55.10592460632324", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751157602219", "lastRunDuration": "45.26185989379883", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:40:01.825550", "end": "2025-06-29 00:40:02.303104", "duration": "0.4775538444519043"}}} 2025-06-29 02:48:43,531 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found check types executed on VR basic, start: 2025-06-29 00:48:02.078967, end: 2025-06-29 00:48:13.886838, duration: 11.807870864868164 2025-06-29 02:48:43,541 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2393311f- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 02:48:02 CEST 2025, details: service is running which took running duration (ms) 14.412164688110352 2025-06-29 02:48:43,554 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@29bdac40- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 02:48:02 CEST 2025, details: service is running which took running duration (ms) 27.24456787109375 2025-06-29 02:48:43,564 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@73778d12- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 02:48:02 CEST 2025, details: service is running which took running duration (ms) 40.70901870727539 2025-06-29 02:48:43,575 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4c572a83- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 02:48:02 CEST 2025, details: service is running which took running duration (ms) 54.00395393371582 2025-06-29 02:48:43,585 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7676e1df- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 02:48:02 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 45.96877098083496 2025-06-29 02:48:43,595 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@32ed8cfd- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 02:48:02 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8101.41921043396 2025-06-29 02:48:43,608 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7dbd98af- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 02:48:10 CEST 2025, details: Memory Usage within limits with current at 50.8774% which took running duration (ms) 71.84505462646484 2025-06-29 02:48:43,617 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@8cbb6a9- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 02:48:10 CEST 2025, details: CPU Usage within limits with current at 0.3% which took running duration (ms) 3234.6138954162598 2025-06-29 02:48:43,627 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@49f2c676- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 02:48:13 CEST 2025, details: Sufficient free space is 2571.59765625 MB which took running duration (ms) 42.53530502319336 2025-06-29 02:48:43,627 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found check types executed on VR advanced, start: 2025-06-29 00:40:01.825550, end: 2025-06-29 00:40:02.303104, duration: 0.4775538444519043 2025-06-29 02:48:43,639 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@17a631d1- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 02:40:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 46.48184776306152 2025-06-29 02:48:43,650 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6cfab6a- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 02:40:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 54.00204658508301 2025-06-29 02:48:43,659 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@53642be4- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 02:40:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 55.10592460632324 2025-06-29 02:48:43,670 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d20140c9]) (logid:671940e0) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@cfa8688- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 02:40:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 45.26185989379883 2025-06-29 02:48:44,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bad5465]) (logid:0d5d7b16) No inactive management server node found 2025-06-29 02:48:44,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bad5465]) (logid:0d5d7b16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:44,466 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) StorageCollector is running... 2025-06-29 02:48:44,472 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:48:44,472 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:48:44,472 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:48:44,473 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-6717118844223095465: Routed from 90520738888109 2025-06-29 02:48:44,527 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 4-6717118844223095465: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:48:44,532 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:48:44,532 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:48:44,532 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:48:44,533 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-1612007191621929901: Routed from 90520738888109 2025-06-29 02:48:44,544 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-1612007191621929901: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:48:44,554 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:48:44,554 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:48:44,554 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:48:44,555 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-3048092522799696626: Routed from 90520738888109 2025-06-29 02:48:44,602 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 1-3048092522799696626: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:48:44,612 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:48:44,612 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:48:44,612 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:48:44,613 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-3048092522799696627: Routed from 90520738888109 2025-06-29 02:48:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:48:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:48:45,580 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2144d313]) (logid:35d20d73) HA health check task is running... 2025-06-29 02:48:45,705 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 1-3048092522799696627: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:48:45,707 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:48:45,707 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:48:45,707 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:48:45,708 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-1612007191621929902: Routed from 90520738888109 2025-06-29 02:48:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8500a186]) (logid:dbef00ab) No inactive management server node found 2025-06-29 02:48:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8500a186]) (logid:dbef00ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-28666309]) (logid:1bbb23ca) VM scheduler.poll is being called at 2025-06-29T00:49:00+0000 2025-06-29 02:48:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-28666309]) (logid:1bbb23ca) Got 0 scheduled jobs to be executed at 2025-06-29T00:49:00+0000 2025-06-29 02:48:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-28666309]) (logid:1bbb23ca) Cleaned up 0 VM scheduled job entries 2025-06-29 02:48:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d0fba080]) (logid:b3ab7b31) VM Operation Thread Running 2025-06-29 02:48:46,177 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-3:[ctx-cd3a92b4]) (logid:675fbd30) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 02:48:46,331 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3f676865]) (logid:b341aba9) ClusterDRS.poll is being called at 2025-06-29T00:49:00+0000 2025-06-29 02:48:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3f676865]) (logid:b341aba9) Removed 0 old drs migration plans 2025-06-29 02:48:46,767 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-7645da95]) (logid:6b3793ad) Seq 2-1612007191621929902: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:48:47,225 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:48:47,226 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:48:47,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:48:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:48:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-843d82a8]) (logid:f20d7348) No inactive management server node found 2025-06-29 02:48:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-843d82a8]) (logid:f20d7348) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1301810]) (logid:1c995c31) No inactive management server node found 2025-06-29 02:48:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1301810]) (logid:1c995c31) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:49,582 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3d5c91d1]) (logid:99055ee6) HA health check task is running... 2025-06-29 02:48:50,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7893: Seq 7-7893: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:48:50,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7893 2025-06-29 02:48:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ed50adb]) (logid:eca0b6cd) No inactive management server node found 2025-06-29 02:48:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ed50adb]) (logid:eca0b6cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:51,029 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-ec57d911]) (logid:e76921d2) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 02:48:51,040 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-ec57d911]) (logid:e76921d2) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 02:48:51,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3ea0aa0]) (logid:231962a6) No inactive management server node found 2025-06-29 02:48:51,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3ea0aa0]) (logid:231962a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:53,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43fb0e9e]) (logid:0d161685) No inactive management server node found 2025-06-29 02:48:53,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43fb0e9e]) (logid:0d161685) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:53,583 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-867883fc]) (logid:df1d22cc) HA health check task is running... 2025-06-29 02:48:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e196ede4]) (logid:fb12b6ed) No inactive management server node found 2025-06-29 02:48:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e196ede4]) (logid:fb12b6ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:56,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 02:48:56,211 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Seq 2-3048092522799696628: Routed from 90520738888109 2025-06-29 02:48:56,340 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-006a6b30]) (logid:249d2300) No inactive management server node found 2025-06-29 02:48:56,340 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-006a6b30]) (logid:249d2300) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-16f4039d]) (logid:fc74e3f8) Enabled non-edge zones available for scan: 1 2025-06-29 02:48:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-16f4039d]) (logid:fc74e3f8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2326 TB], threshold=[90.00%]. 2025-06-29 02:48:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-16f4039d]) (logid:fc74e3f8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:48:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8e0933e5]) (logid:a9b5badd) Enabled non-edge zones available for scan: 1 2025-06-29 02:48:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8e0933e5]) (logid:a9b5badd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:48:57,585 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7fb62d60]) (logid:24bb5283) HA health check task is running... 2025-06-29 02:48:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d48b98f7]) (logid:be9c2274) No inactive management server node found 2025-06-29 02:48:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d48b98f7]) (logid:be9c2274) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:57,873 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-1b57173f]) (logid:cc3ecd6a) AutoScaling Monitor is running... 2025-06-29 02:48:57,971 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-bde51b3c]) (logid:ffd3ec3b) Starting to remove all volume_stats rows older than [Sat Jun 28 14:48:57 CEST 2025]. 2025-06-29 02:48:57,972 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-bde51b3c]) (logid:ffd3ec3b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:48:57 CEST 2025]. 2025-06-29 02:48:59,197 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:48:59,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:48:59,201 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) Seq 2-3048092522799696629: Routed from 90520738888109 2025-06-29 02:48:59,213 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) Seq 1-3048092522799696629: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:48:59,237 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Seq 1-3048092522799696628: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 02:48:59,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 02:48:59,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 02:48:59,268 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Seq 2-1612007191621929903: Routed from 90520738888109 2025-06-29 02:48:59,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:48:59,298 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) Seq 2-1612007191621929904: Routed from 90520738888109 2025-06-29 02:48:59,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb2b3bce]) (logid:df50875f) No inactive management server node found 2025-06-29 02:48:59,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb2b3bce]) (logid:df50875f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:48:59,348 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-5aafcb6f]) (logid:5b735a29) Seq 2-1612007191621929904: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:49:00,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-7894: Seq 7-7894: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:49:00,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 7894 2025-06-29 02:49:00,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-813c500b]) (logid:a7e05fde) No inactive management server node found 2025-06-29 02:49:00,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-813c500b]) (logid:a7e05fde) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:01,586 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-036e463d]) (logid:3ee618b1) HA health check task is running... 2025-06-29 02:49:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1a78a6e]) (logid:78dabda1) No inactive management server node found 2025-06-29 02:49:02,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1a78a6e]) (logid:78dabda1) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faabfafb]) (logid:27d6b015) No inactive management server node found 2025-06-29 02:49:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faabfafb]) (logid:27d6b015) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:04,395 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Seq 2-1612007191621929903: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 02:49:04,395 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-353814f8]) (logid:94379bec) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 02:49:05,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75b15f81]) (logid:5181b897) No inactive management server node found 2025-06-29 02:49:05,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75b15f81]) (logid:5181b897) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:05,587 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-62a7d5d4]) (logid:7e23627b) HA health check task is running... 2025-06-29 02:49:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98ce5278]) (logid:5ff7a14f) No inactive management server node found 2025-06-29 02:49:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98ce5278]) (logid:5ff7a14f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed651edb]) (logid:6b5191bc) No inactive management server node found 2025-06-29 02:49:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed651edb]) (logid:6b5191bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:09,588 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8f96b2d1]) (logid:f3375f21) HA health check task is running... 2025-06-29 02:49:09,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a02c2c7]) (logid:02d86aad) No inactive management server node found 2025-06-29 02:49:09,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a02c2c7]) (logid:02d86aad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:10,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7895: Seq 7-7895: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:49:10,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7895 2025-06-29 02:49:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-23142891]) (logid:d5c142b7) Found 0 routers to update status. 2025-06-29 02:49:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-23142891]) (logid:d5c142b7) Found 0 VPC's to update Redundant State. 2025-06-29 02:49:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-23142891]) (logid:d5c142b7) Found 0 networks to update RvR status. 2025-06-29 02:49:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Running Capacity Checker ... 2025-06-29 02:49:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) recalculating system capacity 2025-06-29 02:49:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Executing cpu/ram capacity update 2025-06-29 02:49:11,069 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:11,070 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:11,077 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 02:49:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 02:49:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 02:49:11,091 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:11,092 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:11,096 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:11,097 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 02:49:11,097 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 02:49:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-308-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 02:49:11,106 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Done executing cpu/ram capacity update 2025-06-29 02:49:11,106 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Executing storage capacity update 2025-06-29 02:49:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:49:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 02:49:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 02:49:11,114 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 02:49:11,119 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:49:11,120 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 02:49:11,120 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 02:49:11,122 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 02:49:11,127 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:49:11,127 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 02:49:11,127 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 02:49:11,131 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 02:49:11,136 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:49:11,136 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 02:49:11,136 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 02:49:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-309-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 02:49:11,140 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Done executing storage capacity update 2025-06-29 02:49:11,140 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Executing capacity updates for public ip and Vlans 2025-06-29 02:49:11,157 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Done capacity updates for public ip and Vlans 2025-06-29 02:49:11,157 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Executing capacity updates for private ip 2025-06-29 02:49:11,160 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Done executing capacity updates for private ip 2025-06-29 02:49:11,160 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Done recalculating system capacity 2025-06-29 02:49:11,187 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-45cf18ea]) (logid:4c3dac0e) Done running Capacity Checker ... 2025-06-29 02:49:11,257 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-f0090ca6]) (logid:215045a5) Checking if any host reservation can be released ... 2025-06-29 02:49:11,262 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-f0090ca6]) (logid:215045a5) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:11,264 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-f0090ca6]) (logid:215045a5) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:11,264 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-f0090ca6]) (logid:215045a5) Done running HostReservationReleaseChecker ... 2025-06-29 02:49:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15ffb4ef]) (logid:321d337e) No inactive management server node found 2025-06-29 02:49:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15ffb4ef]) (logid:321d337e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47659d0c]) (logid:b981f685) No inactive management server node found 2025-06-29 02:49:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47659d0c]) (logid:b981f685) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:12,986 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-bd32ca89]) (logid:196fb0c5) Starting to remove all vm_stats rows older than [Sat Jun 28 14:49:12 CEST 2025]. 2025-06-29 02:49:13,012 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-bd32ca89]) (logid:196fb0c5) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:49:12 CEST 2025]. 2025-06-29 02:49:13,590 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dfb117ec]) (logid:4cc1e499) HA health check task is running... 2025-06-29 02:49:14,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-273eb888]) (logid:5ee1edf9) No inactive management server node found 2025-06-29 02:49:14,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-273eb888]) (logid:5ee1edf9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d448d2a]) (logid:0476c785) No inactive management server node found 2025-06-29 02:49:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d448d2a]) (logid:0476c785) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:17,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-47720e83]) (logid:de343761) Resetting hosts suitable for reconnect 2025-06-29 02:49:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-47720e83]) (logid:de343761) Completed resetting hosts suitable for reconnect 2025-06-29 02:49:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-47720e83]) (logid:de343761) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:49:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-47720e83]) (logid:de343761) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:49:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-47720e83]) (logid:de343761) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:49:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-47720e83]) (logid:de343761) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:49:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60bb697b]) (logid:602c2c80) No inactive management server node found 2025-06-29 02:49:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60bb697b]) (logid:602c2c80) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:17,591 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b5bfe6f6]) (logid:65a9c896) HA health check task is running... 2025-06-29 02:49:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cf80eda]) (logid:dac1cac5) No inactive management server node found 2025-06-29 02:49:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cf80eda]) (logid:dac1cac5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:20,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7896: Seq 7-7896: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:49:20,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7896 2025-06-29 02:49:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4573b24e]) (logid:94b10fc1) No inactive management server node found 2025-06-29 02:49:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4573b24e]) (logid:94b10fc1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:21,592 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5701b5df]) (logid:ef115b17) HA health check task is running... 2025-06-29 02:49:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85aeb56f]) (logid:57e342cc) No inactive management server node found 2025-06-29 02:49:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85aeb56f]) (logid:57e342cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:23,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44f75cac]) (logid:1d18e65c) No inactive management server node found 2025-06-29 02:49:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44f75cac]) (logid:1d18e65c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f73e6ba]) (logid:906f95b5) No inactive management server node found 2025-06-29 02:49:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f73e6ba]) (logid:906f95b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:25,594 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5909bcd7]) (logid:4b485a52) HA health check task is running... 2025-06-29 02:49:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e28f641]) (logid:15b6ff78) No inactive management server node found 2025-06-29 02:49:26,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e28f641]) (logid:15b6ff78) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e4d652c0]) (logid:5e9a91d3) Enabled non-edge zones available for scan: 1 2025-06-29 02:49:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e4d652c0]) (logid:5e9a91d3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2326 TB], threshold=[90.00%]. 2025-06-29 02:49:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e4d652c0]) (logid:5e9a91d3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:49:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-46c1d1c2]) (logid:c0056bb5) Enabled non-edge zones available for scan: 1 2025-06-29 02:49:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-46c1d1c2]) (logid:c0056bb5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:49:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cb4a86f]) (logid:fbe49be9) No inactive management server node found 2025-06-29 02:49:27,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cb4a86f]) (logid:fbe49be9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:29,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f2c7330]) (logid:c8206ec6) No inactive management server node found 2025-06-29 02:49:29,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f2c7330]) (logid:c8206ec6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:29,591 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:49:29,592 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:49:29,594 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) Seq 2-3048092522799696630: Routed from 90520738888109 2025-06-29 02:49:29,595 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f17b23e4]) (logid:678ac287) HA health check task is running... 2025-06-29 02:49:29,636 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) Seq 1-3048092522799696630: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:49:29,637 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:49:29,638 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) Seq 2-1612007191621929905: Routed from 90520738888109 2025-06-29 02:49:29,641 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b0766665]) (logid:801a9888) Seq 2-1612007191621929905: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:49:30,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7897: Seq 7-7897: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:49:30,079 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7897 2025-06-29 02:49:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-073beeca]) (logid:1baf0c9b) No inactive management server node found 2025-06-29 02:49:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-073beeca]) (logid:1baf0c9b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:32,341 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1f8b884]) (logid:dc3f76ee) No inactive management server node found 2025-06-29 02:49:32,341 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1f8b884]) (logid:dc3f76ee) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:33,597 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f0727ca4]) (logid:bbbed2ff) HA health check task is running... 2025-06-29 02:49:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a309b06]) (logid:8ddc0aa2) No inactive management server node found 2025-06-29 02:49:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a309b06]) (logid:8ddc0aa2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:49:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:49:35,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edd6def0]) (logid:a9ca4716) No inactive management server node found 2025-06-29 02:49:35,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edd6def0]) (logid:a9ca4716) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:49:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1721d911]) (logid:2e41c59e) No inactive management server node found 2025-06-29 02:49:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1721d911]) (logid:2e41c59e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:37,598 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d742aa5c]) (logid:756ae2f8) HA health check task is running... 2025-06-29 02:49:38,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-814861cf]) (logid:63446787) No inactive management server node found 2025-06-29 02:49:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-814861cf]) (logid:63446787) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34c7a8d6]) (logid:bd1e36c7) No inactive management server node found 2025-06-29 02:49:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34c7a8d6]) (logid:bd1e36c7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:40,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7898: Seq 7-7898: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:49:40,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7898 2025-06-29 02:49:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b70084a5]) (logid:667505ef) Found 0 routers to update status. 2025-06-29 02:49:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b70084a5]) (logid:667505ef) Found 0 VPC's to update Redundant State. 2025-06-29 02:49:41,005 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b70084a5]) (logid:667505ef) Found 0 networks to update RvR status. 2025-06-29 02:49:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-138613b0]) (logid:c9ec7029) DbCollector is running... 2025-06-29 02:49:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6bc6403]) (logid:d77493a2) No inactive management server node found 2025-06-29 02:49:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6bc6403]) (logid:d77493a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:41,599 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-de18d4ae]) (logid:41d5f67d) HA health check task is running... 2025-06-29 02:49:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) ManagementServerCollector is running... 2025-06-29 02:49:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) DB properties were already loaded 2025-06-29 02:49:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:49:42,171 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Execution of process [4062112] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:49:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Exit value of process [4062112] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:49:42,171 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Process [4062112] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:49:42,171 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) usage install: null 2025-06-29 02:49:42,171 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Sessions active: 0 2025-06-29 02:49:42,173 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:49:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062115] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:49:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:49:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062120] for command [/bin/bash -c uname -r ]. 2025-06-29 02:49:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:49:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062121] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:49:42,191 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) system memory from /proc: 4887515136 2025-06-29 02:49:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:49:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062126] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:49:42,196 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) free memory from /proc: 514883584 2025-06-29 02:49:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:49:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062131] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:49:42,212 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) used memory from /proc: 756084 2025-06-29 02:49:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:49:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062132] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:49:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:49:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062133] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:49:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:49:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062134] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:49:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:49:42,246 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062141] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:49:42,246 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:49:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Successfully executed process [4062142] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail buffersdirect.count, 120 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail buffersdirect.used, 657986 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail buffersmapped.count, 7 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail buffersmapped.used, 798602 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail gcPS-Scavenge.count, 483 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail gcPS-Scavenge.time, 3545 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail jvmuptime, 42122331 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memoryheap.committed, 350224384 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memoryheap.init, 77594624 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memoryheap.usage, 0.10281670037877 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorynon-heap.max, -1 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorynon-heap.usage, -2.20132016E8 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorynon-heap.used, 220132016 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17560942082111436 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.PS-Eden-Space.usage, 0.03040239962206146 2025-06-29 02:49:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.PS-Old-Gen.usage, 0.11846166196562614 2025-06-29 02:49:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorypools.PS-Survivor-Space.usage, 0.5328163146972656 2025-06-29 02:49:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorytotal.committed, 575143936 2025-06-29 02:49:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorytotal.init, 85262336 2025-06-29 02:49:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorytotal.max, 1908932607 2025-06-29 02:49:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail memorytotal.used, 416452712 2025-06-29 02:49:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail threadsdeadlocks, [] 2025-06-29 02:49:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail threadsnew.count, 0 2025-06-29 02:49:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) not storing detail threadstimed_waiting.count, 84 2025-06-29 02:49:42,259 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-ba961975]) (logid:b1ab91af) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:49:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":154708624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42122248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.18,"totalInit":85262336,"totalUsed":415650432,"totalCommitted":575143936,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196270152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":756084,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.18,0.25,0.24],"systemCyclesUsage":[66363118,53803829,4587574425],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:49:42,259 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1403, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:49:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":154708624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42122248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.18,"totalInit":85262336,"totalUsed":415650432,"totalCommitted":575143936,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196270152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":756084,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.18,0.25,0.24],"systemCyclesUsage":[66363118,53803829,4587574425],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:49:42,259 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:49:42,263 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:49:42,263 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-314:[ctx-4c9ee6a5]) (logid:2f00a05b) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:49:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":154708624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42122248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.18,"totalInit":85262336,"totalUsed":415650432,"totalCommitted":575143936,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196270152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":756084,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.18,0.25,0.24],"systemCyclesUsage":[66363118,53803829,4587574425],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:49:42,263 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1403, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:49:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350224384,"freeJvmMemoryBytes":154708624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42122248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.18,"totalInit":85262336,"totalUsed":415650432,"totalCommitted":575143936,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196270152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":756084,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.18,0.25,0.24],"systemCyclesUsage":[66363118,53803829,4587574425],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:49:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76aeedec]) (logid:bdab9286) No inactive management server node found 2025-06-29 02:49:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76aeedec]) (logid:bdab9286) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9268e0da]) (logid:51df7534) No inactive management server node found 2025-06-29 02:49:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9268e0da]) (logid:51df7534) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:45,395 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:49:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:49:45,601 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-08a937b6]) (logid:3f023d45) HA health check task is running... 2025-06-29 02:49:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-70127354]) (logid:573b6fcb) No inactive management server node found 2025-06-29 02:49:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-70127354]) (logid:573b6fcb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-778dd2fe]) (logid:5311f387) VM scheduler.poll is being called at 2025-06-29T00:50:00+0000 2025-06-29 02:49:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-778dd2fe]) (logid:5311f387) Got 0 scheduled jobs to be executed at 2025-06-29T00:50:00+0000 2025-06-29 02:49:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-778dd2fe]) (logid:5311f387) Cleaned up 0 VM scheduled job entries 2025-06-29 02:49:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-3dc03c3a]) (logid:bd8baa98) VM Operation Thread Running 2025-06-29 02:49:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4194c1f2]) (logid:6bc67492) ClusterDRS.poll is being called at 2025-06-29T00:50:00+0000 2025-06-29 02:49:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4194c1f2]) (logid:6bc67492) Removed 0 old drs migration plans 2025-06-29 02:49:46,767 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) StorageCollector is running... 2025-06-29 02:49:46,772 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:49:46,772 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:49:46,772 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:49:46,773 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-6717118844223095466: Routed from 90520738888109 2025-06-29 02:49:46,829 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 4-6717118844223095466: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:49:46,832 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:49:46,832 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:49:46,832 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:49:46,833 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-1612007191621929906: Routed from 90520738888109 2025-06-29 02:49:46,886 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-1612007191621929906: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:49:46,907 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:49:46,908 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:49:46,908 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:49:46,909 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-1612007191621929907: Routed from 90520738888109 2025-06-29 02:49:47,004 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-1612007191621929907: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:49:47,014 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:49:47,014 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:49:47,014 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:49:47,015 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-3048092522799696631: Routed from 90520738888109 2025-06-29 02:49:47,253 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:47,255 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:47,257 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:49:47,259 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:49:47,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f840b5]) (logid:be46c026) No inactive management server node found 2025-06-29 02:49:47,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f840b5]) (logid:be46c026) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:48,071 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 1-3048092522799696631: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:49:48,073 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:49:48,073 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:49:48,073 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:49:48,074 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-1612007191621929908: Routed from 90520738888109 2025-06-29 02:49:48,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a6f380d]) (logid:82b13f3a) No inactive management server node found 2025-06-29 02:49:48,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a6f380d]) (logid:82b13f3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:49,134 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-ee011266]) (logid:b049a6d7) Seq 2-1612007191621929908: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:49:49,602 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5561b1bd]) (logid:2faa324d) HA health check task is running... 2025-06-29 02:49:50,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7900: Seq 7-7900: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:49:50,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7900 2025-06-29 02:49:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40b4bde1]) (logid:681e8e46) No inactive management server node found 2025-06-29 02:49:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40b4bde1]) (logid:681e8e46) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:50,799 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 02:49:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d14981a]) (logid:6bbc96fd) No inactive management server node found 2025-06-29 02:49:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d14981a]) (logid:6bbc96fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71b433ed]) (logid:119d365f) No inactive management server node found 2025-06-29 02:49:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71b433ed]) (logid:119d365f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:53,603 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-279b5a7d]) (logid:b5370f96) HA health check task is running... 2025-06-29 02:49:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af94068e]) (logid:df410b3b) No inactive management server node found 2025-06-29 02:49:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af94068e]) (logid:df410b3b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:49:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8185233]) (logid:82e0eab7) No inactive management server node found 2025-06-29 02:49:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8185233]) (logid:82e0eab7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f2b32455]) (logid:9b629bc7) Enabled non-edge zones available for scan: 1 2025-06-29 02:49:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f2b32455]) (logid:9b629bc7) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2327 TB], threshold=[90.00%]. 2025-06-29 02:49:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f2b32455]) (logid:9b629bc7) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:49:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5358680b]) (logid:e819b868) Enabled non-edge zones available for scan: 1 2025-06-29 02:49:56,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5358680b]) (logid:e819b868) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:49:57,605 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-33c94494]) (logid:b9a7b309) HA health check task is running... 2025-06-29 02:49:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cee4b879]) (logid:4940c44b) No inactive management server node found 2025-06-29 02:49:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cee4b879]) (logid:4940c44b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:49:57,875 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-f54f662f]) (logid:4a94b538) AutoScaling Monitor is running... 2025-06-29 02:49:57,973 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-0157e23b]) (logid:4c1889e1) Starting to remove all volume_stats rows older than [Sat Jun 28 14:49:57 CEST 2025]. 2025-06-29 02:49:57,974 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-0157e23b]) (logid:4c1889e1) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:49:57 CEST 2025]. 2025-06-29 02:49:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8e2c33d]) (logid:054548d4) No inactive management server node found 2025-06-29 02:49:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8e2c33d]) (logid:054548d4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:49:59,404 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:49:59,407 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:49:59,408 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) Seq 2-3048092522799696632: Routed from 90520738888109 2025-06-29 02:49:59,420 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) Seq 1-3048092522799696632: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:49:59,453 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:49:59,455 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) Seq 2-1612007191621929909: Routed from 90520738888109 2025-06-29 02:49:59,502 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-451fee78]) (logid:711f1a13) Seq 2-1612007191621929909: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:50:00,072 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7901: Seq 7-7901: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:00,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7901 2025-06-29 02:50:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99d2744f]) (logid:c30e009e) No inactive management server node found 2025-06-29 02:50:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99d2744f]) (logid:c30e009e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:01,606 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-157ef5d2]) (logid:0fb01759) HA health check task is running... 2025-06-29 02:50:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-970cc917]) (logid:61590d7e) No inactive management server node found 2025-06-29 02:50:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-970cc917]) (logid:61590d7e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:50:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e3ba5e8]) (logid:3f704db6) No inactive management server node found 2025-06-29 02:50:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e3ba5e8]) (logid:3f704db6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:05,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b68c343a]) (logid:716fc11a) No inactive management server node found 2025-06-29 02:50:05,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b68c343a]) (logid:716fc11a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:05,607 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-669ab95a]) (logid:cc29ab7b) HA health check task is running... 2025-06-29 02:50:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf785863]) (logid:5720cffa) No inactive management server node found 2025-06-29 02:50:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf785863]) (logid:5720cffa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bce2531c]) (logid:b92f8650) No inactive management server node found 2025-06-29 02:50:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bce2531c]) (logid:b92f8650) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:09,609 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2bd3b97a]) (logid:39ac5ff9) HA health check task is running... 2025-06-29 02:50:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-907b9a1e]) (logid:f3418013) No inactive management server node found 2025-06-29 02:50:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-907b9a1e]) (logid:f3418013) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:10,072 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7902: Seq 7-7902: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:10,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7902 2025-06-29 02:50:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e2bfbc8c]) (logid:e20c72f3) Found 0 routers to update status. 2025-06-29 02:50:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e2bfbc8c]) (logid:e20c72f3) Found 0 VPC's to update Redundant State. 2025-06-29 02:50:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e2bfbc8c]) (logid:e20c72f3) Found 0 networks to update RvR status. 2025-06-29 02:50:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19982158]) (logid:ae342101) No inactive management server node found 2025-06-29 02:50:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19982158]) (logid:ae342101) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac6b3cb9]) (logid:3a70e411) No inactive management server node found 2025-06-29 02:50:12,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac6b3cb9]) (logid:3a70e411) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:13,014 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-b2185a6c]) (logid:71020737) Starting to remove all vm_stats rows older than [Sat Jun 28 14:50:13 CEST 2025]. 2025-06-29 02:50:13,040 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-b2185a6c]) (logid:71020737) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:50:13 CEST 2025]. 2025-06-29 02:50:13,610 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-120e3f2d]) (logid:32b339fc) HA health check task is running... 2025-06-29 02:50:14,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecffa04b]) (logid:1d19cee0) No inactive management server node found 2025-06-29 02:50:14,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecffa04b]) (logid:1d19cee0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:15,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-913da8a6]) (logid:db6b5503) No inactive management server node found 2025-06-29 02:50:15,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-913da8a6]) (logid:db6b5503) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:17,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f1bc600]) (logid:648ad25b) No inactive management server node found 2025-06-29 02:50:17,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f1bc600]) (logid:648ad25b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:17,611 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c16ceb0e]) (logid:e3306792) HA health check task is running... 2025-06-29 02:50:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4773f86f]) (logid:dc7abb93) No inactive management server node found 2025-06-29 02:50:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4773f86f]) (logid:dc7abb93) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:20,073 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7903: Seq 7-7903: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:20,094 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7903 2025-06-29 02:50:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36fe8ed5]) (logid:bf9ec798) No inactive management server node found 2025-06-29 02:50:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36fe8ed5]) (logid:bf9ec798) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:21,612 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d61dfd96]) (logid:fc1e3159) HA health check task is running... 2025-06-29 02:50:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06bd5d67]) (logid:5dfccc23) No inactive management server node found 2025-06-29 02:50:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06bd5d67]) (logid:5dfccc23) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:23,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4f8bfa0]) (logid:c069d0b9) No inactive management server node found 2025-06-29 02:50:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4f8bfa0]) (logid:c069d0b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:24,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eafc4ea0]) (logid:8727ddad) No inactive management server node found 2025-06-29 02:50:24,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eafc4ea0]) (logid:8727ddad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:25,613 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cfcb6b07]) (logid:0284ac99) HA health check task is running... 2025-06-29 02:50:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0fa82bc]) (logid:7c7e3762) No inactive management server node found 2025-06-29 02:50:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0fa82bc]) (logid:7c7e3762) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-31cf4d43]) (logid:c38a21f5) Enabled non-edge zones available for scan: 1 2025-06-29 02:50:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-31cf4d43]) (logid:c38a21f5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2327 TB], threshold=[90.00%]. 2025-06-29 02:50:26,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-31cf4d43]) (logid:c38a21f5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:50:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1380460f]) (logid:bdaa1b67) Enabled non-edge zones available for scan: 1 2025-06-29 02:50:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1380460f]) (logid:bdaa1b67) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:50:27,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f378982a]) (logid:69f69fd5) No inactive management server node found 2025-06-29 02:50:27,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f378982a]) (logid:69f69fd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:29,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8670bc5]) (logid:1e7546f0) No inactive management server node found 2025-06-29 02:50:29,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8670bc5]) (logid:1e7546f0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:29,615 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-caafd84a]) (logid:62eb71d2) HA health check task is running... 2025-06-29 02:50:29,643 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:50:29,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:50:29,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) Seq 2-3048092522799696633: Routed from 90520738888109 2025-06-29 02:50:29,689 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) Seq 1-3048092522799696633: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:50:29,691 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:50:29,692 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) Seq 2-1612007191621929910: Routed from 90520738888109 2025-06-29 02:50:29,695 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-c5e6e60b]) (logid:2923437e) Seq 2-1612007191621929910: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:50:30,074 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7904: Seq 7-7904: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:30,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7904 2025-06-29 02:50:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63125d75]) (logid:ebbf1aae) No inactive management server node found 2025-06-29 02:50:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63125d75]) (logid:ebbf1aae) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f85f9aef]) (logid:85183c88) No inactive management server node found 2025-06-29 02:50:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f85f9aef]) (logid:85183c88) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:50:33,616 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7c5559b7]) (logid:9b2c9263) HA health check task is running... 2025-06-29 02:50:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c210770]) (logid:838e9b02) No inactive management server node found 2025-06-29 02:50:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c210770]) (logid:838e9b02) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:33,985 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:50:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:50:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:50:33,989 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:50:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-319dd83e]) (logid:bd2e0182) No inactive management server node found 2025-06-29 02:50:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-319dd83e]) (logid:bd2e0182) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f06a31f]) (logid:550720f8) No inactive management server node found 2025-06-29 02:50:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f06a31f]) (logid:550720f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:37,617 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-111deedb]) (logid:f3218769) HA health check task is running... 2025-06-29 02:50:38,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba4c7d51]) (logid:5b23de00) No inactive management server node found 2025-06-29 02:50:38,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba4c7d51]) (logid:5b23de00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e3ab901]) (logid:6fcd14cd) No inactive management server node found 2025-06-29 02:50:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e3ab901]) (logid:6fcd14cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:40,074 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7905: Seq 7-7905: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:40,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7905 2025-06-29 02:50:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ecfa0e97]) (logid:6ea0d787) Found 0 routers to update status. 2025-06-29 02:50:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ecfa0e97]) (logid:6ea0d787) Found 0 VPC's to update Redundant State. 2025-06-29 02:50:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ecfa0e97]) (logid:6ea0d787) Found 0 networks to update RvR status. 2025-06-29 02:50:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-71fdb1b2]) (logid:10acd189) DbCollector is running... 2025-06-29 02:50:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283fce7c]) (logid:ee209fc7) No inactive management server node found 2025-06-29 02:50:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283fce7c]) (logid:ee209fc7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:50:41,618 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7cd78e18]) (logid:6e3d34ef) HA health check task is running... 2025-06-29 02:50:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) ManagementServerCollector is running... 2025-06-29 02:50:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) DB properties were already loaded 2025-06-29 02:50:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:50:42,170 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Execution of process [4062157] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:50:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Exit value of process [4062157] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:50:42,170 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Process [4062157] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:50:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) usage install: null 2025-06-29 02:50:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Sessions active: 0 2025-06-29 02:50:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:50:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062160] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:50:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:50:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062165] for command [/bin/bash -c uname -r ]. 2025-06-29 02:50:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:50:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062166] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:50:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) system memory from /proc: 4887515136 2025-06-29 02:50:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:50:42,199 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062171] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:50:42,200 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) free memory from /proc: 510783488 2025-06-29 02:50:42,200 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:50:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062176] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:50:42,214 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) used memory from /proc: 755956 2025-06-29 02:50:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:50:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062177] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:50:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:50:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062178] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:50:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:50:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062179] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:50:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:50:42,253 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062186] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:50:42,253 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:50:42,260 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Successfully executed process [4062187] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail buffersdirect.count, 120 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail buffersdirect.used, 657986 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail buffersmapped.count, 7 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail buffersmapped.used, 798602 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail gcPS-Scavenge.count, 484 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail gcPS-Scavenge.time, 3550 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail jvmuptime, 42182340 2025-06-29 02:50:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memoryheap.committed, 350748672 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memoryheap.init, 77594624 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memoryheap.usage, 0.1010223237802222 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorynon-heap.max, -1 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorynon-heap.usage, -2.20132016E8 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorynon-heap.used, 220132016 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17560942082111436 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.PS-Eden-Space.usage, 0.025421915321032926 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.PS-Old-Gen.usage, 0.11850171139806848 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorypools.PS-Survivor-Space.usage, 0.5059574672154018 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorytotal.committed, 575668224 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorytotal.init, 85262336 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorytotal.max, 1908932607 2025-06-29 02:50:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail memorytotal.used, 413044144 2025-06-29 02:50:42,264 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail threadsdeadlocks, [] 2025-06-29 02:50:42,264 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail threadsnew.count, 0 2025-06-29 02:50:42,265 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) not storing detail threadstimed_waiting.count, 84 2025-06-29 02:50:42,268 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-c81355c1]) (logid:86616404) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:50:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":158651832,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42182247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":412228856,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192844808,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510783488,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.25,0.24],"systemCyclesUsage":[66363486,53804166,4587597473],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:50:42,268 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1405, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:50:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":158651832,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42182247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":412228856,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192844808,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510783488,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.25,0.24],"systemCyclesUsage":[66363486,53804166,4587597473],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:50:42,268 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:50:42,272 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:50:42,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1405, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:50:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":158651832,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42182247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":412228856,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192844808,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510783488,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.25,0.24],"systemCyclesUsage":[66363486,53804166,4587597473],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:50:42,272 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-315:[ctx-258f27f2]) (logid:a174a7ae) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:50:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":350748672,"freeJvmMemoryBytes":158651832,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42182247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":412228856,"totalCommitted":575668224,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192844808,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510783488,"systemMemoryUsed":755956,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.25,0.24],"systemCyclesUsage":[66363486,53804166,4587597473],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:50:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fe5bb8d]) (logid:dfba1a01) No inactive management server node found 2025-06-29 02:50:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fe5bb8d]) (logid:dfba1a01) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:44,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d33a36d]) (logid:822df73c) No inactive management server node found 2025-06-29 02:50:44,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d33a36d]) (logid:822df73c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:50:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:50:45,619 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ce3498dc]) (logid:55ec4822) HA health check task is running... 2025-06-29 02:50:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39d0351b]) (logid:addbe3a4) No inactive management server node found 2025-06-29 02:50:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39d0351b]) (logid:addbe3a4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cab4a255]) (logid:96422cdf) VM scheduler.poll is being called at 2025-06-29T00:51:00+0000 2025-06-29 02:50:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cab4a255]) (logid:96422cdf) Got 0 scheduled jobs to be executed at 2025-06-29T00:51:00+0000 2025-06-29 02:50:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cab4a255]) (logid:96422cdf) Cleaned up 0 VM scheduled job entries 2025-06-29 02:50:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-2ad7b685]) (logid:0232e829) VM Operation Thread Running 2025-06-29 02:50:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-560f6a6f]) (logid:9bb481d2) ClusterDRS.poll is being called at 2025-06-29T00:51:00+0000 2025-06-29 02:50:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-560f6a6f]) (logid:9bb481d2) Removed 0 old drs migration plans 2025-06-29 02:50:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c18708f]) (logid:b5f763bf) Resetting hosts suitable for reconnect 2025-06-29 02:50:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c18708f]) (logid:b5f763bf) Completed resetting hosts suitable for reconnect 2025-06-29 02:50:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c18708f]) (logid:b5f763bf) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:50:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c18708f]) (logid:b5f763bf) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:50:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c18708f]) (logid:b5f763bf) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:50:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c18708f]) (logid:b5f763bf) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:50:47,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:50:47,215 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:50:47,217 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:50:47,220 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:50:47,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69cd88dc]) (logid:46497407) No inactive management server node found 2025-06-29 02:50:47,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69cd88dc]) (logid:46497407) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cde956ca]) (logid:c8baff98) No inactive management server node found 2025-06-29 02:50:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cde956ca]) (logid:c8baff98) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:49,143 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) StorageCollector is running... 2025-06-29 02:50:49,149 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:50:49,149 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:50:49,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:50:49,150 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 2-6717118844223095467: Routed from 90520738888109 2025-06-29 02:50:49,204 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 4-6717118844223095467: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:50:49,209 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:50:49,209 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:50:49,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:50:49,210 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 2-3048092522799696634: Routed from 90520738888109 2025-06-29 02:50:49,258 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 1-3048092522799696634: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:50:49,282 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:50:49,282 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:50:49,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:50:49,283 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 2-3048092522799696635: Routed from 90520738888109 2025-06-29 02:50:49,371 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 1-3048092522799696635: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:50:49,381 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:50:49,381 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:50:49,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:50:49,382 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 2-3048092522799696636: Routed from 90520738888109 2025-06-29 02:50:49,620 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-738c5fd6]) (logid:d0e308bd) HA health check task is running... 2025-06-29 02:50:50,074 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7907: Seq 7-7907: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:50,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7907 2025-06-29 02:50:50,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1224d26]) (logid:c9405b4e) No inactive management server node found 2025-06-29 02:50:50,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1224d26]) (logid:c9405b4e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:50,474 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 1-3048092522799696636: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:50:50,476 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:50:50,476 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:50:50,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:50:50,477 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 2-1612007191621929911: Routed from 90520738888109 2025-06-29 02:50:51,537 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-084c54d9]) (logid:fcd18d23) Seq 2-1612007191621929911: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:50:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ede113de]) (logid:76cf3246) No inactive management server node found 2025-06-29 02:50:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ede113de]) (logid:76cf3246) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:53,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d043ec8e]) (logid:e624686c) No inactive management server node found 2025-06-29 02:50:53,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d043ec8e]) (logid:e624686c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:53,622 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e8f11069]) (logid:ef456a05) HA health check task is running... 2025-06-29 02:50:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cacb97bf]) (logid:8f02f8fc) No inactive management server node found 2025-06-29 02:50:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cacb97bf]) (logid:8f02f8fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:55,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7908: Seq 7-7908: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:50:55,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7908 2025-06-29 02:50:56,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1c58565]) (logid:373f4176) No inactive management server node found 2025-06-29 02:50:56,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1c58565]) (logid:373f4176) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-48878cce]) (logid:18db421b) Enabled non-edge zones available for scan: 1 2025-06-29 02:50:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-48878cce]) (logid:18db421b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2328 TB], threshold=[90.00%]. 2025-06-29 02:50:56,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-48878cce]) (logid:18db421b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:50:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ebab3823]) (logid:b2afb211) Enabled non-edge zones available for scan: 1 2025-06-29 02:50:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ebab3823]) (logid:b2afb211) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:50:57,624 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e2c326f9]) (logid:9b99f14f) HA health check task is running... 2025-06-29 02:50:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-573c5b94]) (logid:cf192a65) No inactive management server node found 2025-06-29 02:50:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-573c5b94]) (logid:cf192a65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:57,877 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-44f9fbed]) (logid:ed567672) AutoScaling Monitor is running... 2025-06-29 02:50:57,982 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-606ce19b]) (logid:30433baf) Starting to remove all volume_stats rows older than [Sat Jun 28 14:50:57 CEST 2025]. 2025-06-29 02:50:57,982 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-606ce19b]) (logid:30433baf) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:50:57 CEST 2025]. 2025-06-29 02:50:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e43568f]) (logid:21754c9a) No inactive management server node found 2025-06-29 02:50:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e43568f]) (logid:21754c9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:50:59,562 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:50:59,565 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:50:59,566 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) Seq 2-3048092522799696637: Routed from 90520738888109 2025-06-29 02:50:59,579 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) Seq 1-3048092522799696637: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:50:59,611 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:50:59,612 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) Seq 2-1612007191621929912: Routed from 90520738888109 2025-06-29 02:50:59,663 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-84ff1d66]) (logid:585cec75) Seq 2-1612007191621929912: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:51:00,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50cd38a4]) (logid:94192014) No inactive management server node found 2025-06-29 02:51:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50cd38a4]) (logid:94192014) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:01,625 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f9b89791]) (logid:e6b4548f) HA health check task is running... 2025-06-29 02:51:02,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2387a96]) (logid:ea6e6db5) No inactive management server node found 2025-06-29 02:51:02,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2387a96]) (logid:ea6e6db5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:03,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4faefdec]) (logid:8ea98fea) No inactive management server node found 2025-06-29 02:51:03,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4faefdec]) (logid:8ea98fea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:05,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7909: Seq 7-7909: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:51:05,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7909 2025-06-29 02:51:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee8a15c5]) (logid:6f681bd9) No inactive management server node found 2025-06-29 02:51:05,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee8a15c5]) (logid:6f681bd9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:05,626 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-00b64392]) (logid:4a84f1e6) HA health check task is running... 2025-06-29 02:51:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0c17632]) (logid:36eab19c) No inactive management server node found 2025-06-29 02:51:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0c17632]) (logid:36eab19c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:08,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d93adf1d]) (logid:8898a43d) No inactive management server node found 2025-06-29 02:51:08,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d93adf1d]) (logid:8898a43d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:09,627 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3d8a17fa]) (logid:e62cf24f) HA health check task is running... 2025-06-29 02:51:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a689922f]) (logid:5d052d14) No inactive management server node found 2025-06-29 02:51:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a689922f]) (logid:5d052d14) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a5af3b44]) (logid:a90e56d5) Found 0 routers to update status. 2025-06-29 02:51:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a5af3b44]) (logid:a90e56d5) Found 0 VPC's to update Redundant State. 2025-06-29 02:51:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a5af3b44]) (logid:a90e56d5) Found 0 networks to update RvR status. 2025-06-29 02:51:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c237035e]) (logid:7a7e8799) No inactive management server node found 2025-06-29 02:51:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c237035e]) (logid:7a7e8799) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09155157]) (logid:2bdea79b) No inactive management server node found 2025-06-29 02:51:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09155157]) (logid:2bdea79b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:13,041 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-493684da]) (logid:b5f8142a) Starting to remove all vm_stats rows older than [Sat Jun 28 14:51:13 CEST 2025]. 2025-06-29 02:51:13,062 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-493684da]) (logid:b5f8142a) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:51:13 CEST 2025]. 2025-06-29 02:51:13,628 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d99e0c72]) (logid:d8588fae) HA health check task is running... 2025-06-29 02:51:14,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-266d2f8e]) (logid:4d9eb375) No inactive management server node found 2025-06-29 02:51:14,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-266d2f8e]) (logid:4d9eb375) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:15,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7910: Seq 7-7910: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:51:15,085 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7910 2025-06-29 02:51:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a4a43c9]) (logid:b0033581) No inactive management server node found 2025-06-29 02:51:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a4a43c9]) (logid:b0033581) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cdd7209]) (logid:0413dabd) No inactive management server node found 2025-06-29 02:51:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cdd7209]) (logid:0413dabd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:17,629 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3c41c4e3]) (logid:59866be6) HA health check task is running... 2025-06-29 02:51:18,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9621b17]) (logid:6ff35064) No inactive management server node found 2025-06-29 02:51:18,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9621b17]) (logid:6ff35064) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:20,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab7e49af]) (logid:c19e0834) No inactive management server node found 2025-06-29 02:51:20,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab7e49af]) (logid:c19e0834) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:21,631 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ae122757]) (logid:4da7abcc) HA health check task is running... 2025-06-29 02:51:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bf4927e]) (logid:82918ea4) No inactive management server node found 2025-06-29 02:51:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bf4927e]) (logid:82918ea4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:23,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1530d6c5]) (logid:6ec12139) No inactive management server node found 2025-06-29 02:51:23,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1530d6c5]) (logid:6ec12139) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:24,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3bbf6f8]) (logid:ec688159) No inactive management server node found 2025-06-29 02:51:24,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3bbf6f8]) (logid:ec688159) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:25,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7911: Seq 7-7911: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:51:25,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7911 2025-06-29 02:51:25,632 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-322bf063]) (logid:f15f7f3a) HA health check task is running... 2025-06-29 02:51:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-614dcde2]) (logid:22fa8df2) No inactive management server node found 2025-06-29 02:51:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-614dcde2]) (logid:22fa8df2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:26,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-80263b78]) (logid:33933a45) Enabled non-edge zones available for scan: 1 2025-06-29 02:51:26,764 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-80263b78]) (logid:33933a45) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2328 TB], threshold=[90.00%]. 2025-06-29 02:51:26,766 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-80263b78]) (logid:33933a45) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:51:26,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-08bcc675]) (logid:32cfb1f0) Enabled non-edge zones available for scan: 1 2025-06-29 02:51:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-08bcc675]) (logid:32cfb1f0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:51:27,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92b67d93]) (logid:0ae33ae9) No inactive management server node found 2025-06-29 02:51:27,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92b67d93]) (logid:0ae33ae9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:29,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-621cb6f7]) (logid:b70aa700) No inactive management server node found 2025-06-29 02:51:29,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-621cb6f7]) (logid:b70aa700) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:29,633 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9e25b84e]) (logid:c4ddc1b0) HA health check task is running... 2025-06-29 02:51:29,697 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:51:29,698 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:51:29,699 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) Seq 2-3048092522799696638: Routed from 90520738888109 2025-06-29 02:51:29,742 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) Seq 1-3048092522799696638: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:51:29,743 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:51:29,744 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) Seq 2-1612007191621929913: Routed from 90520738888109 2025-06-29 02:51:29,747 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e0bd9678]) (logid:351ff0fe) Seq 2-1612007191621929913: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:51:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fff9ec51]) (logid:50625765) No inactive management server node found 2025-06-29 02:51:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fff9ec51]) (logid:50625765) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99803d91]) (logid:f08a06b4) No inactive management server node found 2025-06-29 02:51:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99803d91]) (logid:f08a06b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:33,634 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c9ce96c9]) (logid:31e926bf) HA health check task is running... 2025-06-29 02:51:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aebf7ffa]) (logid:45fc9708) No inactive management server node found 2025-06-29 02:51:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aebf7ffa]) (logid:45fc9708) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:51:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:51:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:51:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:51:35,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7912: Seq 7-7912: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:51:35,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7912 2025-06-29 02:51:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-388a72e1]) (logid:af0dd868) No inactive management server node found 2025-06-29 02:51:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-388a72e1]) (logid:af0dd868) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9461d8b]) (logid:09ad5e4d) No inactive management server node found 2025-06-29 02:51:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9461d8b]) (logid:09ad5e4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:37,636 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-03bfa74c]) (logid:08246d39) HA health check task is running... 2025-06-29 02:51:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65fc114e]) (logid:8c3ee33b) No inactive management server node found 2025-06-29 02:51:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65fc114e]) (logid:8c3ee33b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d639794d]) (logid:ff685c03) No inactive management server node found 2025-06-29 02:51:39,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d639794d]) (logid:ff685c03) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fd8ffec8]) (logid:3215d7e0) Found 0 routers to update status. 2025-06-29 02:51:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fd8ffec8]) (logid:3215d7e0) Found 0 VPC's to update Redundant State. 2025-06-29 02:51:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fd8ffec8]) (logid:3215d7e0) Found 0 networks to update RvR status. 2025-06-29 02:51:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-a2dad084]) (logid:7278529d) DbCollector is running... 2025-06-29 02:51:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b25e07]) (logid:fca1adbe) No inactive management server node found 2025-06-29 02:51:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b25e07]) (logid:fca1adbe) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:51:41,637 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-30611ada]) (logid:882fc67f) HA health check task is running... 2025-06-29 02:51:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) ManagementServerCollector is running... 2025-06-29 02:51:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) DB properties were already loaded 2025-06-29 02:51:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:51:42,167 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Execution of process [4062193] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:51:42,167 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Exit value of process [4062193] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:51:42,167 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Process [4062193] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:51:42,167 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) usage install: null 2025-06-29 02:51:42,167 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Sessions active: 0 2025-06-29 02:51:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:51:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062196] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:51:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:51:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062201] for command [/bin/bash -c uname -r ]. 2025-06-29 02:51:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:51:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062202] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:51:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) system memory from /proc: 4887515136 2025-06-29 02:51:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:51:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062207] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:51:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) free memory from /proc: 522051584 2025-06-29 02:51:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:51:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062212] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:51:42,208 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) used memory from /proc: 758004 2025-06-29 02:51:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:51:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062213] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:51:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:51:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062214] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:51:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:51:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062215] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:51:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:51:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062222] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:51:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:51:42,250 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Successfully executed process [4062223] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail buffersdirect.count, 120 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail buffersdirect.used, 657986 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail buffersmapped.count, 7 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail buffersmapped.used, 798602 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail gcPS-Scavenge.count, 485 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail gcPS-Scavenge.time, 3554 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail jvmuptime, 42242329 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memoryheap.committed, 351272960 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memoryheap.init, 77594624 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memoryheap.usage, 0.09928869736191337 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorynon-heap.max, -1 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorynon-heap.usage, -2.20132016E8 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorynon-heap.used, 220132016 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17560942082111436 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.PS-Eden-Space.usage, 0.02065509499155565 2025-06-29 02:51:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.PS-Old-Gen.usage, 0.11866190912783779 2025-06-29 02:51:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorypools.PS-Survivor-Space.usage, 0.48512413388206843 2025-06-29 02:51:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorytotal.committed, 576192512 2025-06-29 02:51:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorytotal.init, 85262336 2025-06-29 02:51:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorytotal.max, 1908932607 2025-06-29 02:51:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail memorytotal.used, 409722320 2025-06-29 02:51:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail threadsdeadlocks, [] 2025-06-29 02:51:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail threadsnew.count, 0 2025-06-29 02:51:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:51:42,256 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-7189e536]) (logid:e32b87c0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:51:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":162489176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42242245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":408915800,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189535432,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522051584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.27,0.24],"systemCyclesUsage":[66363804,53804474,4587620596],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:51:42,256 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1407, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:51:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":162489176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42242245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":408915800,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189535432,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522051584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.27,0.24],"systemCyclesUsage":[66363804,53804474,4587620596],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:51:42,256 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:51:42,261 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:51:42,262 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1407, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:51:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":162489176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42242245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":408915800,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189535432,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522051584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.27,0.24],"systemCyclesUsage":[66363804,53804474,4587620596],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:51:42,262 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-315:[ctx-e7b1fbe1]) (logid:14f22642) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:51:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":162489176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42242245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":408915800,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189535432,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":522051584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.27,0.24],"systemCyclesUsage":[66363804,53804474,4587620596],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:51:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01eb9911]) (logid:fc782f01) No inactive management server node found 2025-06-29 02:51:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01eb9911]) (logid:fc782f01) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21e48154]) (logid:01f622ab) No inactive management server node found 2025-06-29 02:51:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21e48154]) (logid:01f622ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:45,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7913: Seq 7-7913: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:51:45,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7913 2025-06-29 02:51:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:51:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:51:45,638 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5e35f959]) (logid:e924be73) HA health check task is running... 2025-06-29 02:51:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d30830ae]) (logid:7c3346b8) No inactive management server node found 2025-06-29 02:51:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d30830ae]) (logid:7c3346b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d29e3bc8]) (logid:142a0178) VM scheduler.poll is being called at 2025-06-29T00:52:00+0000 2025-06-29 02:51:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d29e3bc8]) (logid:142a0178) Got 0 scheduled jobs to be executed at 2025-06-29T00:52:00+0000 2025-06-29 02:51:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d29e3bc8]) (logid:142a0178) Cleaned up 0 VM scheduled job entries 2025-06-29 02:51:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-c91b46c6]) (logid:2ba8f0e3) VM Operation Thread Running 2025-06-29 02:51:46,331 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-92647974]) (logid:ed0603f8) ClusterDRS.poll is being called at 2025-06-29T00:52:00+0000 2025-06-29 02:51:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-92647974]) (logid:ed0603f8) Removed 0 old drs migration plans 2025-06-29 02:51:47,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:51:47,253 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:51:47,254 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:51:47,256 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:51:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73d0b3f0]) (logid:286a25dc) No inactive management server node found 2025-06-29 02:51:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73d0b3f0]) (logid:286a25dc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-150ec847]) (logid:2060a51c) No inactive management server node found 2025-06-29 02:51:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-150ec847]) (logid:2060a51c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:49,639 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-308d933c]) (logid:757ca53b) HA health check task is running... 2025-06-29 02:51:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd253357]) (logid:aaf26c8b) No inactive management server node found 2025-06-29 02:51:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd253357]) (logid:aaf26c8b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:51,538 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) StorageCollector is running... 2025-06-29 02:51:51,543 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:51:51,544 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:51:51,544 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:51:51,544 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-6717118844223095468: Routed from 90520738888109 2025-06-29 02:51:51,598 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 4-6717118844223095468: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:51:51,603 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:51:51,603 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:51:51,603 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:51:51,604 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-1612007191621929914: Routed from 90520738888109 2025-06-29 02:51:51,655 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-1612007191621929914: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:51:51,677 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:51:51,677 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:51:51,677 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:51:51,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-1612007191621929915: Routed from 90520738888109 2025-06-29 02:51:51,770 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-1612007191621929915: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:51:51,780 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:51:51,781 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:51:51,781 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:51:51,782 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-3048092522799696639: Routed from 90520738888109 2025-06-29 02:51:51,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3edee25a]) (logid:6c79d83b) No inactive management server node found 2025-06-29 02:51:51,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3edee25a]) (logid:6c79d83b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:52,832 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 1-3048092522799696639: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:51:52,834 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:51:52,834 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:51:52,834 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:51:52,835 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-1612007191621929916: Routed from 90520738888109 2025-06-29 02:51:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a558cf6d]) (logid:83f50554) No inactive management server node found 2025-06-29 02:51:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a558cf6d]) (logid:83f50554) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:53,641 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b4c43515]) (logid:41959afb) HA health check task is running... 2025-06-29 02:51:53,894 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8c965e21]) (logid:1b6a48a1) Seq 2-1612007191621929916: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:51:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64acc598]) (logid:aa5fed64) No inactive management server node found 2025-06-29 02:51:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64acc598]) (logid:aa5fed64) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:55,078 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7915: Seq 7-7915: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:51:55,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7915 2025-06-29 02:51:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edd17b14]) (logid:66414093) No inactive management server node found 2025-06-29 02:51:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edd17b14]) (logid:66414093) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-45970392]) (logid:8db7488b) Enabled non-edge zones available for scan: 1 2025-06-29 02:51:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-45970392]) (logid:8db7488b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 02:51:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-45970392]) (logid:8db7488b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:51:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9f0c0438]) (logid:ddf31f41) Enabled non-edge zones available for scan: 1 2025-06-29 02:51:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9f0c0438]) (logid:ddf31f41) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:51:57,642 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-bf08b821]) (logid:7b7d8b9e) HA health check task is running... 2025-06-29 02:51:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b76efe3a]) (logid:e5918b16) No inactive management server node found 2025-06-29 02:51:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b76efe3a]) (logid:e5918b16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:51:57,879 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-47509957]) (logid:1802fe32) AutoScaling Monitor is running... 2025-06-29 02:51:57,984 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-4060ebba]) (logid:0c9aac32) Starting to remove all volume_stats rows older than [Sat Jun 28 14:51:57 CEST 2025]. 2025-06-29 02:51:57,985 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-4060ebba]) (logid:0c9aac32) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:51:57 CEST 2025]. 2025-06-29 02:51:59,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5733a7d3]) (logid:1ea092dc) No inactive management server node found 2025-06-29 02:51:59,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5733a7d3]) (logid:1ea092dc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:51:59,719 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:51:59,722 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:51:59,723 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) Seq 2-3048092522799696640: Routed from 90520738888109 2025-06-29 02:51:59,736 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) Seq 1-3048092522799696640: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:51:59,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:51:59,786 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) Seq 2-1612007191621929917: Routed from 90520738888109 2025-06-29 02:51:59,830 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-598c9bcd]) (logid:36865576) Seq 2-1612007191621929917: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:52:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60429039]) (logid:5fbeb1d2) No inactive management server node found 2025-06-29 02:52:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60429039]) (logid:5fbeb1d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:01,643 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1bd5bfea]) (logid:9f50606a) HA health check task is running... 2025-06-29 02:52:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c61800a]) (logid:bbf5b5d7) No inactive management server node found 2025-06-29 02:52:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c61800a]) (logid:bbf5b5d7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb83805b]) (logid:5a8c996a) No inactive management server node found 2025-06-29 02:52:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb83805b]) (logid:5a8c996a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:05,078 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7916: Seq 7-7916: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:52:05,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7916 2025-06-29 02:52:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3954d965]) (logid:42f98597) No inactive management server node found 2025-06-29 02:52:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3954d965]) (logid:42f98597) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:05,645 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c909439e]) (logid:af78fee9) HA health check task is running... 2025-06-29 02:52:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a8eac38]) (logid:a41048b4) No inactive management server node found 2025-06-29 02:52:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a8eac38]) (logid:a41048b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67725f34]) (logid:9fb30d34) No inactive management server node found 2025-06-29 02:52:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67725f34]) (logid:9fb30d34) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:09,646 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7a349522]) (logid:7b14222e) HA health check task is running... 2025-06-29 02:52:09,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-669cbce5]) (logid:cbd34ba4) No inactive management server node found 2025-06-29 02:52:09,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-669cbce5]) (logid:cbd34ba4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fc7b2824]) (logid:f4f8feac) Found 0 routers to update status. 2025-06-29 02:52:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fc7b2824]) (logid:f4f8feac) Found 0 VPC's to update Redundant State. 2025-06-29 02:52:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fc7b2824]) (logid:f4f8feac) Found 0 networks to update RvR status. 2025-06-29 02:52:11,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99681dab]) (logid:dbc84ba9) No inactive management server node found 2025-06-29 02:52:11,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99681dab]) (logid:dbc84ba9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:52:12,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed2e708c]) (logid:be4cda28) No inactive management server node found 2025-06-29 02:52:12,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed2e708c]) (logid:be4cda28) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:13,064 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-dd9df5c1]) (logid:32c84d87) Starting to remove all vm_stats rows older than [Sat Jun 28 14:52:13 CEST 2025]. 2025-06-29 02:52:13,088 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-dd9df5c1]) (logid:32c84d87) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:52:13 CEST 2025]. 2025-06-29 02:52:13,647 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3b2a8870]) (logid:a580319f) HA health check task is running... 2025-06-29 02:52:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de118509]) (logid:57e4eba4) No inactive management server node found 2025-06-29 02:52:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de118509]) (logid:57e4eba4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:15,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-7917: Seq 7-7917: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:52:15,106 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 7917 2025-06-29 02:52:15,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e79f0966]) (logid:fae68252) No inactive management server node found 2025-06-29 02:52:15,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e79f0966]) (logid:fae68252) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7a2f29f8]) (logid:560cd589) Resetting hosts suitable for reconnect 2025-06-29 02:52:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7a2f29f8]) (logid:560cd589) Completed resetting hosts suitable for reconnect 2025-06-29 02:52:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7a2f29f8]) (logid:560cd589) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:52:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7a2f29f8]) (logid:560cd589) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:52:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7a2f29f8]) (logid:560cd589) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:52:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7a2f29f8]) (logid:560cd589) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:52:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-712ae473]) (logid:db635289) No inactive management server node found 2025-06-29 02:52:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-712ae473]) (logid:db635289) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:17,648 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-031673b1]) (logid:cbbb6020) HA health check task is running... 2025-06-29 02:52:18,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bafc6d6]) (logid:586b095d) No inactive management server node found 2025-06-29 02:52:18,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bafc6d6]) (logid:586b095d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:20,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b923a15]) (logid:4227015b) No inactive management server node found 2025-06-29 02:52:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b923a15]) (logid:4227015b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:21,650 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-18ffad95]) (logid:4e8ac5ff) HA health check task is running... 2025-06-29 02:52:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cdd8ce4]) (logid:54c83648) No inactive management server node found 2025-06-29 02:52:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cdd8ce4]) (logid:54c83648) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:23,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4f3827f]) (logid:da89e66d) No inactive management server node found 2025-06-29 02:52:23,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4f3827f]) (logid:da89e66d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:24,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e68f2963]) (logid:decb396c) No inactive management server node found 2025-06-29 02:52:24,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e68f2963]) (logid:decb396c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:25,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7918: Seq 7-7918: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:52:25,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7918 2025-06-29 02:52:25,651 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-65301162]) (logid:4555cf18) HA health check task is running... 2025-06-29 02:52:26,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b028fe88]) (logid:845c4b2b) No inactive management server node found 2025-06-29 02:52:26,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b028fe88]) (logid:845c4b2b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f1508901]) (logid:144bd845) Enabled non-edge zones available for scan: 1 2025-06-29 02:52:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f1508901]) (logid:144bd845) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 02:52:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f1508901]) (logid:144bd845) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:52:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fefb597f]) (logid:4ca009ca) Enabled non-edge zones available for scan: 1 2025-06-29 02:52:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fefb597f]) (logid:4ca009ca) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:52:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-521bd1f3]) (logid:437329d5) No inactive management server node found 2025-06-29 02:52:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-521bd1f3]) (logid:437329d5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:29,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d328bfbf]) (logid:9c44f288) No inactive management server node found 2025-06-29 02:52:29,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d328bfbf]) (logid:9c44f288) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:29,652 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b45dbf41]) (logid:fb08119b) HA health check task is running... 2025-06-29 02:52:29,750 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:52:29,751 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:52:29,753 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) Seq 2-3048092522799696641: Routed from 90520738888109 2025-06-29 02:52:29,796 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) Seq 1-3048092522799696641: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:52:29,797 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:52:29,798 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) Seq 2-1612007191621929918: Routed from 90520738888109 2025-06-29 02:52:29,801 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-69d45b4a]) (logid:a360f748) Seq 2-1612007191621929918: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:52:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bb2b168]) (logid:092be286) No inactive management server node found 2025-06-29 02:52:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bb2b168]) (logid:092be286) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb9bff17]) (logid:98cdf19e) No inactive management server node found 2025-06-29 02:52:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb9bff17]) (logid:98cdf19e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:52:33,654 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cf34e071]) (logid:0e65af28) HA health check task is running... 2025-06-29 02:52:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d1ef4cc]) (logid:f033a5e9) No inactive management server node found 2025-06-29 02:52:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d1ef4cc]) (logid:f033a5e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:52:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:52:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:52:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:52:35,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7919: Seq 7-7919: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:52:35,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7919 2025-06-29 02:52:35,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-051c090f]) (logid:1f488e94) No inactive management server node found 2025-06-29 02:52:35,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-051c090f]) (logid:1f488e94) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5108055]) (logid:fe3c80c2) No inactive management server node found 2025-06-29 02:52:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5108055]) (logid:fe3c80c2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:37,655 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e83f620c]) (logid:a813c454) HA health check task is running... 2025-06-29 02:52:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47388e98]) (logid:d75538b8) No inactive management server node found 2025-06-29 02:52:38,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47388e98]) (logid:d75538b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:39,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0875ccab]) (logid:22d2ac81) No inactive management server node found 2025-06-29 02:52:39,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0875ccab]) (logid:22d2ac81) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4d6a7a07]) (logid:ce39102e) Found 0 routers to update status. 2025-06-29 02:52:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4d6a7a07]) (logid:ce39102e) Found 0 VPC's to update Redundant State. 2025-06-29 02:52:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4d6a7a07]) (logid:ce39102e) Found 0 networks to update RvR status. 2025-06-29 02:52:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-d3fd5eb2]) (logid:99880757) DbCollector is running... 2025-06-29 02:52:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-745a22e6]) (logid:57526467) No inactive management server node found 2025-06-29 02:52:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-745a22e6]) (logid:57526467) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:41,656 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-feee5674]) (logid:4f369620) HA health check task is running... 2025-06-29 02:52:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) ManagementServerCollector is running... 2025-06-29 02:52:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) DB properties were already loaded 2025-06-29 02:52:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:52:42,169 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Execution of process [4062236] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:52:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Exit value of process [4062236] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:52:42,169 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Process [4062236] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:52:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) usage install: null 2025-06-29 02:52:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Sessions active: 0 2025-06-29 02:52:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:52:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062239] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:52:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:52:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062244] for command [/bin/bash -c uname -r ]. 2025-06-29 02:52:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:52:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062245] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:52:42,186 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) system memory from /proc: 4887515136 2025-06-29 02:52:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:52:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062250] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:52:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) free memory from /proc: 514883584 2025-06-29 02:52:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:52:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062255] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:52:42,204 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) used memory from /proc: 758004 2025-06-29 02:52:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:52:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062256] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:52:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:52:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062257] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:52:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:52:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062258] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:52:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:52:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062265] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:52:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:52:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Successfully executed process [4062266] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail buffersdirect.count, 120 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail buffersdirect.used, 657986 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail buffersmapped.count, 7 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail buffersmapped.used, 798602 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail gcPS-Scavenge.count, 486 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail gcPS-Scavenge.time, 3559 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail jvmuptime, 42302319 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memoryheap.committed, 351272960 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memoryheap.init, 77594624 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memoryheap.usage, 0.09730838648862349 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorynon-heap.max, -1 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorynon-heap.usage, -2.20138544E8 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorynon-heap.used, 220138544 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17566253165822449 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.PS-Eden-Space.usage, 0.015126461709013878 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.PS-Old-Gen.usage, 0.11900384903652508 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorypools.PS-Survivor-Space.usage, 0.44345238095238093 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorytotal.committed, 576192512 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorytotal.init, 85262336 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorytotal.max, 1908932607 2025-06-29 02:52:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail memorytotal.used, 405935248 2025-06-29 02:52:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail threadsdeadlocks, [] 2025-06-29 02:52:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail threadsnew.count, 0 2025-06-29 02:52:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:52:42,246 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-db646c7f]) (logid:c614e36b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:52:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":166286296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42302246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":405145984,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":185755152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.21,0.22],"systemCyclesUsage":[66364129,53804791,4587643693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:52:42,246 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1409, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:52:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":166286296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42302246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":405145984,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":185755152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.21,0.22],"systemCyclesUsage":[66364129,53804791,4587643693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:52:42,247 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:52:42,250 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-315:[ctx-95438703]) (logid:c7a93132) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:52:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":166286296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42302246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":405145984,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":185755152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.21,0.22],"systemCyclesUsage":[66364129,53804791,4587643693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:52:42,251 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:52:42,251 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1409, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:52:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":166286296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42302246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":405145984,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":185755152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":514883584,"systemMemoryUsed":758004,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.21,0.22],"systemCyclesUsage":[66364129,53804791,4587643693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:52:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bd0788b]) (logid:c558d75f) No inactive management server node found 2025-06-29 02:52:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bd0788b]) (logid:c558d75f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42c2da56]) (logid:78457044) No inactive management server node found 2025-06-29 02:52:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42c2da56]) (logid:78457044) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:45,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7920: Seq 7-7920: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:52:45,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7920 2025-06-29 02:52:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:52:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:52:45,658 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c2b7e313]) (logid:358f89d9) HA health check task is running... 2025-06-29 02:52:45,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dfdf12b]) (logid:aaca4503) No inactive management server node found 2025-06-29 02:52:45,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dfdf12b]) (logid:aaca4503) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-4bb2e1ce]) (logid:809d31e9) VM scheduler.poll is being called at 2025-06-29T00:53:00+0000 2025-06-29 02:52:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-4bb2e1ce]) (logid:809d31e9) Got 0 scheduled jobs to be executed at 2025-06-29T00:53:00+0000 2025-06-29 02:52:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-4bb2e1ce]) (logid:809d31e9) Cleaned up 0 VM scheduled job entries 2025-06-29 02:52:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-a478f72b]) (logid:fc09eade) VM Operation Thread Running 2025-06-29 02:52:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5a7da889]) (logid:2e34c4b3) ClusterDRS.poll is being called at 2025-06-29T00:53:00+0000 2025-06-29 02:52:46,337 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5a7da889]) (logid:2e34c4b3) Removed 0 old drs migration plans 2025-06-29 02:52:47,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:52:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:52:47,231 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:52:47,234 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:52:47,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dece588e]) (logid:a9d190f0) No inactive management server node found 2025-06-29 02:52:47,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dece588e]) (logid:a9d190f0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:52:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3981558]) (logid:0c2ebce4) No inactive management server node found 2025-06-29 02:52:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3981558]) (logid:0c2ebce4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:49,659 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-fa10d508]) (logid:5f021d77) HA health check task is running... 2025-06-29 02:52:50,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65169669]) (logid:cfdecf19) No inactive management server node found 2025-06-29 02:52:50,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65169669]) (logid:cfdecf19) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:52:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84bf17d6]) (logid:5312e21d) No inactive management server node found 2025-06-29 02:52:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84bf17d6]) (logid:5312e21d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:53,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5eb8938]) (logid:6aee7b8b) No inactive management server node found 2025-06-29 02:52:53,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5eb8938]) (logid:6aee7b8b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:52:53,660 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-936858b2]) (logid:6444d437) HA health check task is running... 2025-06-29 02:52:53,894 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) StorageCollector is running... 2025-06-29 02:52:53,900 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:52:53,900 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:52:53,900 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:52:53,901 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-6717118844223095469: Routed from 90520738888109 2025-06-29 02:52:53,956 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 4-6717118844223095469: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:52:53,960 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:52:53,960 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:52:53,960 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:52:53,961 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-3048092522799696642: Routed from 90520738888109 2025-06-29 02:52:54,011 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 1-3048092522799696642: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:52:54,034 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:52:54,034 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:52:54,034 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:52:54,035 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-1612007191621929919: Routed from 90520738888109 2025-06-29 02:52:54,087 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-1612007191621929919: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:52:54,097 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:52:54,097 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:52:54,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:52:54,097 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-3048092522799696643: Routed from 90520738888109 2025-06-29 02:52:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05838a00]) (logid:9de55a96) No inactive management server node found 2025-06-29 02:52:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05838a00]) (logid:9de55a96) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:55,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7922: Seq 7-7922: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:52:55,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7922 2025-06-29 02:52:55,149 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 1-3048092522799696643: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:52:55,150 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:52:55,150 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:52:55,150 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:52:55,151 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-1612007191621929920: Routed from 90520738888109 2025-06-29 02:52:56,212 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-38b1d209]) (logid:7c16b8db) Seq 2-1612007191621929920: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:52:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d4b28cd]) (logid:9927bcf9) No inactive management server node found 2025-06-29 02:52:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d4b28cd]) (logid:9927bcf9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:52:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-92854004]) (logid:d0522a9d) Enabled non-edge zones available for scan: 1 2025-06-29 02:52:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-92854004]) (logid:d0522a9d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2334 TB], threshold=[90.00%]. 2025-06-29 02:52:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-92854004]) (logid:d0522a9d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:52:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2a4f0885]) (logid:949c32d9) Enabled non-edge zones available for scan: 1 2025-06-29 02:52:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2a4f0885]) (logid:949c32d9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:52:57,661 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ed90b1b1]) (logid:28d0f799) HA health check task is running... 2025-06-29 02:52:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67317bb7]) (logid:ebcecbd8) No inactive management server node found 2025-06-29 02:52:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67317bb7]) (logid:ebcecbd8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:57,881 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-2be6a221]) (logid:3ccd45db) AutoScaling Monitor is running... 2025-06-29 02:52:57,987 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-c90baf77]) (logid:f907dfaf) Starting to remove all volume_stats rows older than [Sat Jun 28 14:52:57 CEST 2025]. 2025-06-29 02:52:57,987 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-c90baf77]) (logid:f907dfaf) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:52:57 CEST 2025]. 2025-06-29 02:52:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2844971b]) (logid:2bdd90e4) No inactive management server node found 2025-06-29 02:52:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2844971b]) (logid:2bdd90e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:52:59,891 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:52:59,895 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:52:59,897 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) Seq 2-3048092522799696644: Routed from 90520738888109 2025-06-29 02:52:59,908 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) Seq 1-3048092522799696644: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:52:59,941 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:52:59,942 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) Seq 2-1612007191621929921: Routed from 90520738888109 2025-06-29 02:52:59,994 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4a702e9d]) (logid:b020cef6) Seq 2-1612007191621929921: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:53:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccbd2191]) (logid:fc094e9d) No inactive management server node found 2025-06-29 02:53:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccbd2191]) (logid:fc094e9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:01,662 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-02b5f8ea]) (logid:0deaccbf) HA health check task is running... 2025-06-29 02:53:02,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9af9891b]) (logid:ef51408a) No inactive management server node found 2025-06-29 02:53:02,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9af9891b]) (logid:ef51408a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c9b4b34]) (logid:1769fb18) No inactive management server node found 2025-06-29 02:53:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c9b4b34]) (logid:1769fb18) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:05,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7923: Seq 7-7923: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:53:05,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7923 2025-06-29 02:53:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d3ce4fd]) (logid:b04c7562) No inactive management server node found 2025-06-29 02:53:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d3ce4fd]) (logid:b04c7562) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:05,664 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6a380a60]) (logid:0c5d2784) HA health check task is running... 2025-06-29 02:53:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-938c10f5]) (logid:ffe882ea) No inactive management server node found 2025-06-29 02:53:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-938c10f5]) (logid:ffe882ea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:08,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75beb929]) (logid:7558f88e) No inactive management server node found 2025-06-29 02:53:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75beb929]) (logid:7558f88e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:53:09,666 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-860ea94a]) (logid:64904e19) HA health check task is running... 2025-06-29 02:53:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b2904bb]) (logid:87487344) No inactive management server node found 2025-06-29 02:53:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b2904bb]) (logid:87487344) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d0954c1d]) (logid:1b970393) Found 0 routers to update status. 2025-06-29 02:53:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d0954c1d]) (logid:1b970393) Found 0 VPC's to update Redundant State. 2025-06-29 02:53:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d0954c1d]) (logid:1b970393) Found 0 networks to update RvR status. 2025-06-29 02:53:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-386228f2]) (logid:65f47ebf) No inactive management server node found 2025-06-29 02:53:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-386228f2]) (logid:65f47ebf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9c1352a]) (logid:862206bd) No inactive management server node found 2025-06-29 02:53:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9c1352a]) (logid:862206bd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:13,090 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-94e9ac62]) (logid:d32204fe) Starting to remove all vm_stats rows older than [Sat Jun 28 14:53:13 CEST 2025]. 2025-06-29 02:53:13,112 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-94e9ac62]) (logid:d32204fe) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:53:13 CEST 2025]. 2025-06-29 02:53:13,667 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ec0b0f63]) (logid:03bfa700) HA health check task is running... 2025-06-29 02:53:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6c5d222]) (logid:36f8d683) No inactive management server node found 2025-06-29 02:53:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6c5d222]) (logid:36f8d683) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:15,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7924: Seq 7-7924: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:53:15,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7924 2025-06-29 02:53:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b50b659d]) (logid:38658a5c) No inactive management server node found 2025-06-29 02:53:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b50b659d]) (logid:38658a5c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:17,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07425e2f]) (logid:3167b430) No inactive management server node found 2025-06-29 02:53:17,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07425e2f]) (logid:3167b430) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:53:17,668 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-14afc54b]) (logid:32c35e6d) HA health check task is running... 2025-06-29 02:53:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cf02332]) (logid:c665f332) No inactive management server node found 2025-06-29 02:53:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cf02332]) (logid:c665f332) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f081f34]) (logid:448d0bfd) No inactive management server node found 2025-06-29 02:53:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f081f34]) (logid:448d0bfd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:21,669 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d17591ff]) (logid:8f882560) HA health check task is running... 2025-06-29 02:53:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7058e724]) (logid:d0bb56ea) No inactive management server node found 2025-06-29 02:53:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7058e724]) (logid:d0bb56ea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:23,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d1f9796]) (logid:86baba61) No inactive management server node found 2025-06-29 02:53:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d1f9796]) (logid:86baba61) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-408c08b5]) (logid:3b0143b8) No inactive management server node found 2025-06-29 02:53:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-408c08b5]) (logid:3b0143b8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:25,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7925: Seq 7-7925: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:53:25,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7925 2025-06-29 02:53:25,671 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8f0b87c1]) (logid:f29c4a66) HA health check task is running... 2025-06-29 02:53:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2225ae77]) (logid:8788fa2a) No inactive management server node found 2025-06-29 02:53:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2225ae77]) (logid:8788fa2a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:53:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4ab1afdf]) (logid:3180bd65) Enabled non-edge zones available for scan: 1 2025-06-29 02:53:26,766 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4ab1afdf]) (logid:3180bd65) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2334 TB], threshold=[90.00%]. 2025-06-29 02:53:26,768 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4ab1afdf]) (logid:3180bd65) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:53:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5e6f3c31]) (logid:a8ecf1ec) Enabled non-edge zones available for scan: 1 2025-06-29 02:53:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5e6f3c31]) (logid:a8ecf1ec) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:53:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59926d30]) (logid:d3eaa499) No inactive management server node found 2025-06-29 02:53:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59926d30]) (logid:d3eaa499) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:53:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-195ecee2]) (logid:e9bcf9af) No inactive management server node found 2025-06-29 02:53:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-195ecee2]) (logid:e9bcf9af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:29,672 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fa39be3a]) (logid:46dbd543) HA health check task is running... 2025-06-29 02:53:29,804 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:53:29,805 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:53:29,805 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) Seq 2-3048092522799696645: Routed from 90520738888109 2025-06-29 02:53:29,848 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) Seq 1-3048092522799696645: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:53:29,849 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:53:29,850 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) Seq 2-1612007191621929922: Routed from 90520738888109 2025-06-29 02:53:29,853 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-3c7e4a8c]) (logid:9cb8e6e0) Seq 2-1612007191621929922: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:53:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c84a2c11]) (logid:092b4817) No inactive management server node found 2025-06-29 02:53:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c84a2c11]) (logid:092b4817) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efb0121a]) (logid:bc0c3aaa) No inactive management server node found 2025-06-29 02:53:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efb0121a]) (logid:bc0c3aaa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:33,673 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9c16ae31]) (logid:ba0f9a35) HA health check task is running... 2025-06-29 02:53:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af815d60]) (logid:3da2383b) No inactive management server node found 2025-06-29 02:53:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af815d60]) (logid:3da2383b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:33,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:53:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:53:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:53:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:53:35,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7926: Seq 7-7926: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:53:35,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7926 2025-06-29 02:53:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67fb663c]) (logid:145a94cc) No inactive management server node found 2025-06-29 02:53:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67fb663c]) (logid:145a94cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fb8aa2f]) (logid:4ccc272f) No inactive management server node found 2025-06-29 02:53:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fb8aa2f]) (logid:4ccc272f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:37,674 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c1ad6eb6]) (logid:fffe6a7a) HA health check task is running... 2025-06-29 02:53:38,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9d76a3f]) (logid:373426c0) No inactive management server node found 2025-06-29 02:53:38,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9d76a3f]) (logid:373426c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95b9bc61]) (logid:b985cc54) No inactive management server node found 2025-06-29 02:53:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95b9bc61]) (logid:b985cc54) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:40,815 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-d639a9ae]) (logid:78f935ba) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 02:53:40,816 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-d639a9ae]) (logid:78f935ba) Snapshot scheduler is being called at [2025-06-29T00:53:40+0000]. 2025-06-29 02:53:40,816 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-d639a9ae]) (logid:78f935ba) There are [0] scheduled snapshots to be executed at [2025-06-29T00:53:40+0000]. 2025-06-29 02:53:40,916 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-6e767acb]) (logid:61f2ec4e) External devices are not used. Skipping external device usage collection 2025-06-29 02:53:40,978 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-72869656]) (logid:c7e72a1f) Backup backup.poll is being called at 2025-06-29T00:53:40+0000 2025-06-29 02:53:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-dac6f0a3]) (logid:6a1c3760) Found 0 running routers. 2025-06-29 02:53:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0a60e75a]) (logid:67e4e396) Found 0 routers to update status. 2025-06-29 02:53:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0a60e75a]) (logid:67e4e396) Found 0 VPC's to update Redundant State. 2025-06-29 02:53:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0a60e75a]) (logid:67e4e396) Found 0 networks to update RvR status. 2025-06-29 02:53:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-0c74881d]) (logid:d137608c) DbCollector is running... 2025-06-29 02:53:41,277 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-2694543f]) (logid:dad292bb) Started resource counters recalculation periodic task. 2025-06-29 02:53:41,280 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-2694543f]) (logid:dad292bb) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 02:53:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1daedb4e]) (logid:afdbaebb) No inactive management server node found 2025-06-29 02:53:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1daedb4e]) (logid:afdbaebb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:41,625 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-2694543f]) (logid:dad292bb) Finished resource counters recalculation periodic task. 2025-06-29 02:53:41,676 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f6d5551a]) (logid:13e9b363) HA health check task is running... 2025-06-29 02:53:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) ManagementServerCollector is running... 2025-06-29 02:53:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) DB properties were already loaded 2025-06-29 02:53:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:53:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Execution of process [4062271] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:53:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Exit value of process [4062271] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:53:42,170 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Process [4062271] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:53:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) usage install: null 2025-06-29 02:53:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Sessions active: 0 2025-06-29 02:53:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:53:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062274] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:53:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:53:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062279] for command [/bin/bash -c uname -r ]. 2025-06-29 02:53:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:53:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062280] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:53:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) system memory from /proc: 4887515136 2025-06-29 02:53:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:53:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062285] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:53:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) free memory from /proc: 505786368 2025-06-29 02:53:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:53:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062290] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:53:42,211 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) used memory from /proc: 761588 2025-06-29 02:53:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:53:42,218 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062291] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:53:42,218 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:53:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062292] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:53:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:53:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062293] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:53:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:53:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062300] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:53:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:53:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Successfully executed process [4062301] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail buffersdirect.count, 120 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail buffersdirect.used, 657986 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail buffersmapped.count, 7 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail buffersmapped.used, 798602 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail gcPS-Scavenge.count, 487 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail gcPS-Scavenge.time, 3563 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail jvmuptime, 42362327 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memoryheap.committed, 351272960 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memoryheap.init, 77594624 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memoryheap.usage, 0.10077965832516178 2025-06-29 02:53:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorynon-heap.max, -1 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorynon-heap.usage, -2.20139312E8 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorynon-heap.used, 220139312 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17566877999200214 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.PS-Eden-Space.usage, 0.024693695644162735 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.PS-Old-Gen.usage, 0.11934712988603076 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorypools.PS-Survivor-Space.usage, 0.41875 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorytotal.committed, 576192512 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorytotal.init, 85262336 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorytotal.max, 1908932607 2025-06-29 02:53:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail memorytotal.used, 412593552 2025-06-29 02:53:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail threadsdeadlocks, [] 2025-06-29 02:53:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail threadsnew.count, 0 2025-06-29 02:53:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) not storing detail threadstimed_waiting.count, 84 2025-06-29 02:53:42,255 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-2a83f629]) (logid:4fcff454) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:53:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":159639744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42362247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":411772528,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192381576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":505786368,"systemMemoryUsed":761588,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.17,0.21],"systemCyclesUsage":[66364478,53805111,4587666762],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:53:42,255 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1411, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:53:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":159639744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42362247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":411772528,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192381576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":505786368,"systemMemoryUsed":761588,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.17,0.21],"systemCyclesUsage":[66364478,53805111,4587666762],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:53:42,255 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:53:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:53:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1411, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:53:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":159639744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42362247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":411772528,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192381576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":505786368,"systemMemoryUsed":761588,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.17,0.21],"systemCyclesUsage":[66364478,53805111,4587666762],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:53:42,260 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-316:[ctx-d199b6a5]) (logid:4c0e4669) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:53:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":159639744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42362247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":411772528,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192381576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":505786368,"systemMemoryUsed":761588,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.17,0.21],"systemCyclesUsage":[66364478,53805111,4587666762],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:53:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47379607]) (logid:47440373) No inactive management server node found 2025-06-29 02:53:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47379607]) (logid:47440373) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:44,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84ad1ae7]) (logid:a7a6eec5) No inactive management server node found 2025-06-29 02:53:44,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84ad1ae7]) (logid:a7a6eec5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:45,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7927: Seq 7-7927: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:53:45,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7927 2025-06-29 02:53:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:53:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:53:45,677 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4e6b52c5]) (logid:d13d2cef) HA health check task is running... 2025-06-29 02:53:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b462530]) (logid:70d3d6cd) No inactive management server node found 2025-06-29 02:53:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b462530]) (logid:70d3d6cd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-26089960]) (logid:82e02d53) VM scheduler.poll is being called at 2025-06-29T00:54:00+0000 2025-06-29 02:53:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-26089960]) (logid:82e02d53) Got 0 scheduled jobs to be executed at 2025-06-29T00:54:00+0000 2025-06-29 02:53:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-26089960]) (logid:82e02d53) Cleaned up 0 VM scheduled job entries 2025-06-29 02:53:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-2f6296cf]) (logid:c19ae632) VM Operation Thread Running 2025-06-29 02:53:46,179 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-1:[ctx-e426a9a0]) (logid:e52d22ba) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 02:53:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-cdd67b4a]) (logid:f085a8d8) ClusterDRS.poll is being called at 2025-06-29T00:54:00+0000 2025-06-29 02:53:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-cdd67b4a]) (logid:f085a8d8) Removed 0 old drs migration plans 2025-06-29 02:53:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933d5866]) (logid:6bba5505) Resetting hosts suitable for reconnect 2025-06-29 02:53:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933d5866]) (logid:6bba5505) Completed resetting hosts suitable for reconnect 2025-06-29 02:53:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933d5866]) (logid:6bba5505) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:53:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933d5866]) (logid:6bba5505) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:53:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933d5866]) (logid:6bba5505) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:53:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933d5866]) (logid:6bba5505) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:53:47,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:53:47,226 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:53:47,227 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:53:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:53:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d31da6]) (logid:c7f5221d) No inactive management server node found 2025-06-29 02:53:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d31da6]) (logid:c7f5221d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b65491f]) (logid:af657d45) No inactive management server node found 2025-06-29 02:53:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b65491f]) (logid:af657d45) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:49,678 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d5acbba3]) (logid:a353c590) HA health check task is running... 2025-06-29 02:53:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39efadc1]) (logid:91fc8495) No inactive management server node found 2025-06-29 02:53:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39efadc1]) (logid:91fc8495) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:51,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7158b989]) (logid:78cf24ff) No inactive management server node found 2025-06-29 02:53:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7158b989]) (logid:78cf24ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64320c93]) (logid:e2244b34) No inactive management server node found 2025-06-29 02:53:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64320c93]) (logid:e2244b34) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:53,679 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6922d492]) (logid:dea1152f) HA health check task is running... 2025-06-29 02:53:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ab79608]) (logid:c05ee669) No inactive management server node found 2025-06-29 02:53:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ab79608]) (logid:c05ee669) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:55,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7929: Seq 7-7929: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:53:55,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7929 2025-06-29 02:53:56,212 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) StorageCollector is running... 2025-06-29 02:53:56,219 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:53:56,219 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:53:56,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:53:56,222 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 2-6717118844223095470: Routed from 90520738888109 2025-06-29 02:53:56,275 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 4-6717118844223095470: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:53:56,279 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:53:56,279 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:53:56,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:53:56,280 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 2-3048092522799696646: Routed from 90520738888109 2025-06-29 02:53:56,327 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 1-3048092522799696646: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:53:56,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b5bc789]) (logid:81b5e53f) No inactive management server node found 2025-06-29 02:53:56,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b5bc789]) (logid:81b5e53f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:53:56,342 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:53:56,342 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:53:56,342 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:53:56,343 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 2-3048092522799696647: Routed from 90520738888109 2025-06-29 02:53:56,431 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 1-3048092522799696647: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:53:56,443 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:53:56,443 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:53:56,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:53:56,445 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 2-3048092522799696648: Routed from 90520738888109 2025-06-29 02:53:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3247d776]) (logid:9b79c749) Enabled non-edge zones available for scan: 1 2025-06-29 02:53:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3247d776]) (logid:9b79c749) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:53:56,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3247d776]) (logid:9b79c749) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:53:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fe052e0a]) (logid:cfeafc7d) Enabled non-edge zones available for scan: 1 2025-06-29 02:53:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fe052e0a]) (logid:cfeafc7d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:53:57,535 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 1-3048092522799696648: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:53:57,537 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:53:57,537 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:53:57,537 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:53:57,538 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 2-1612007191621929923: Routed from 90520738888109 2025-06-29 02:53:57,681 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b633633c]) (logid:016a4d33) HA health check task is running... 2025-06-29 02:53:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d290607]) (logid:42e98a05) No inactive management server node found 2025-06-29 02:53:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d290607]) (logid:42e98a05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:53:57,883 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-85e03d75]) (logid:188c1d27) AutoScaling Monitor is running... 2025-06-29 02:53:57,989 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-445bc552]) (logid:565eba46) Starting to remove all volume_stats rows older than [Sat Jun 28 14:53:57 CEST 2025]. 2025-06-29 02:53:57,990 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-445bc552]) (logid:565eba46) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:53:57 CEST 2025]. 2025-06-29 02:53:58,596 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3e042025]) (logid:b1f39c9f) Seq 2-1612007191621929923: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:53:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c91d2a0b]) (logid:83ada1f1) No inactive management server node found 2025-06-29 02:53:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c91d2a0b]) (logid:83ada1f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:00,054 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:54:00,056 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:54:00,057 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) Seq 2-3048092522799696649: Routed from 90520738888109 2025-06-29 02:54:00,073 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) Seq 1-3048092522799696649: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:54:00,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7930: Seq 7-7930: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:54:00,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7930 2025-06-29 02:54:00,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:54:00,131 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) Seq 2-1612007191621929924: Routed from 90520738888109 2025-06-29 02:54:00,187 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6380a276]) (logid:f0574389) Seq 2-1612007191621929924: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:54:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2ff788e]) (logid:ea189979) No inactive management server node found 2025-06-29 02:54:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2ff788e]) (logid:ea189979) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:01,682 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1e8e8fea]) (logid:bf5b195f) HA health check task is running... 2025-06-29 02:54:02,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f836d704]) (logid:67f39ab7) No inactive management server node found 2025-06-29 02:54:02,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f836d704]) (logid:67f39ab7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:03,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9c2bde2]) (logid:eb9b5583) No inactive management server node found 2025-06-29 02:54:03,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9c2bde2]) (logid:eb9b5583) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:05,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35335094]) (logid:d19145b5) No inactive management server node found 2025-06-29 02:54:05,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35335094]) (logid:d19145b5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:05,683 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1cb1793a]) (logid:b51b4fd8) HA health check task is running... 2025-06-29 02:54:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fda3d6ec]) (logid:d52f74cd) No inactive management server node found 2025-06-29 02:54:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fda3d6ec]) (logid:d52f74cd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7305eb00]) (logid:b1dd7eeb) No inactive management server node found 2025-06-29 02:54:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7305eb00]) (logid:b1dd7eeb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:09,685 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-10380b4f]) (logid:78d0b950) HA health check task is running... 2025-06-29 02:54:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dd6001d]) (logid:3ed089e7) No inactive management server node found 2025-06-29 02:54:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dd6001d]) (logid:3ed089e7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:10,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7931: Seq 7-7931: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:54:10,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7931 2025-06-29 02:54:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-600f915b]) (logid:177392f8) Found 0 routers to update status. 2025-06-29 02:54:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-600f915b]) (logid:177392f8) Found 0 VPC's to update Redundant State. 2025-06-29 02:54:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-600f915b]) (logid:177392f8) Found 0 networks to update RvR status. 2025-06-29 02:54:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Running Capacity Checker ... 2025-06-29 02:54:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) recalculating system capacity 2025-06-29 02:54:11,065 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Executing cpu/ram capacity update 2025-06-29 02:54:11,069 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:11,070 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:11,076 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 02:54:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 02:54:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 02:54:11,092 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:11,093 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:11,097 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 02:54:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 02:54:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-310-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 02:54:11,106 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Done executing cpu/ram capacity update 2025-06-29 02:54:11,106 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Executing storage capacity update 2025-06-29 02:54:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:54:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 02:54:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 02:54:11,115 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 02:54:11,119 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:54:11,119 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 02:54:11,119 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 02:54:11,122 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 02:54:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:54:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 02:54:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 02:54:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 02:54:11,134 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:54:11,134 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 02:54:11,134 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 02:54:11,137 DEBUG [c.c.s.StorageManagerImpl] (pool-311-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 02:54:11,138 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Done executing storage capacity update 2025-06-29 02:54:11,138 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Executing capacity updates for public ip and Vlans 2025-06-29 02:54:11,156 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Done capacity updates for public ip and Vlans 2025-06-29 02:54:11,156 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Executing capacity updates for private ip 2025-06-29 02:54:11,161 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Done executing capacity updates for private ip 2025-06-29 02:54:11,161 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Done recalculating system capacity 2025-06-29 02:54:11,188 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-e1eccc5c]) (logid:2c50d02b) Done running Capacity Checker ... 2025-06-29 02:54:11,258 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-2dbc09a4]) (logid:e8198975) Checking if any host reservation can be released ... 2025-06-29 02:54:11,262 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-2dbc09a4]) (logid:e8198975) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:11,265 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-2dbc09a4]) (logid:e8198975) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:11,265 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-2dbc09a4]) (logid:e8198975) Done running HostReservationReleaseChecker ... 2025-06-29 02:54:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15feccc1]) (logid:898bde29) No inactive management server node found 2025-06-29 02:54:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15feccc1]) (logid:898bde29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9505173c]) (logid:6cb9008d) No inactive management server node found 2025-06-29 02:54:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9505173c]) (logid:6cb9008d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:13,113 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-4ab911c7]) (logid:9b50bbfc) Starting to remove all vm_stats rows older than [Sat Jun 28 14:54:13 CEST 2025]. 2025-06-29 02:54:13,137 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-4ab911c7]) (logid:9b50bbfc) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:54:13 CEST 2025]. 2025-06-29 02:54:13,686 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-547f1375]) (logid:4f4039f2) HA health check task is running... 2025-06-29 02:54:14,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5b59b2b]) (logid:a225c1ad) No inactive management server node found 2025-06-29 02:54:14,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5b59b2b]) (logid:a225c1ad) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:15,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6610302f]) (logid:75e37f02) No inactive management server node found 2025-06-29 02:54:15,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6610302f]) (logid:75e37f02) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1a4a8ff]) (logid:1012aacf) No inactive management server node found 2025-06-29 02:54:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1a4a8ff]) (logid:1012aacf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:17,687 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-af6ed582]) (logid:671b4d23) HA health check task is running... 2025-06-29 02:54:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99411d44]) (logid:f47ac7c3) No inactive management server node found 2025-06-29 02:54:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99411d44]) (logid:f47ac7c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:20,085 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7932: Seq 7-7932: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:54:20,106 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7932 2025-06-29 02:54:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ef65c76]) (logid:76dc6479) No inactive management server node found 2025-06-29 02:54:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ef65c76]) (logid:76dc6479) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:21,688 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-68deee3e]) (logid:ea999997) HA health check task is running... 2025-06-29 02:54:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-971ab77c]) (logid:6446a100) No inactive management server node found 2025-06-29 02:54:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-971ab77c]) (logid:6446a100) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12828efe]) (logid:05729f51) No inactive management server node found 2025-06-29 02:54:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12828efe]) (logid:05729f51) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01dcc520]) (logid:2aa4be5c) No inactive management server node found 2025-06-29 02:54:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01dcc520]) (logid:2aa4be5c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:25,690 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-93fd3b06]) (logid:a6f85dce) HA health check task is running... 2025-06-29 02:54:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2735b788]) (logid:f1a4bf36) No inactive management server node found 2025-06-29 02:54:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2735b788]) (logid:f1a4bf36) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-535b3cb4]) (logid:e7ae11a2) Enabled non-edge zones available for scan: 1 2025-06-29 02:54:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-535b3cb4]) (logid:e7ae11a2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:54:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-535b3cb4]) (logid:e7ae11a2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:54:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a09c103a]) (logid:c067a192) Enabled non-edge zones available for scan: 1 2025-06-29 02:54:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a09c103a]) (logid:c067a192) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:54:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-472e3341]) (logid:dc154f14) No inactive management server node found 2025-06-29 02:54:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-472e3341]) (logid:dc154f14) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f83bd5d7]) (logid:68a263e6) No inactive management server node found 2025-06-29 02:54:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f83bd5d7]) (logid:68a263e6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:29,691 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-07f2d994]) (logid:7be2afa5) HA health check task is running... 2025-06-29 02:54:29,855 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:54:29,857 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:54:29,859 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) Seq 2-3048092522799696650: Routed from 90520738888109 2025-06-29 02:54:29,901 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) Seq 1-3048092522799696650: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:54:29,902 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:54:29,903 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) Seq 2-1612007191621929925: Routed from 90520738888109 2025-06-29 02:54:29,906 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-edb48b45]) (logid:51235a65) Seq 2-1612007191621929925: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:54:30,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7933: Seq 7-7933: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:54:30,099 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7933 2025-06-29 02:54:30,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a599b106]) (logid:a485c214) No inactive management server node found 2025-06-29 02:54:30,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a599b106]) (logid:a485c214) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:32,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36e85ca5]) (logid:6b792053) No inactive management server node found 2025-06-29 02:54:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36e85ca5]) (logid:6b792053) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:54:33,692 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-89f3b7d3]) (logid:0ad3ed57) HA health check task is running... 2025-06-29 02:54:33,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2168f30d]) (logid:6c469fd6) No inactive management server node found 2025-06-29 02:54:33,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2168f30d]) (logid:6c469fd6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:33,991 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:33,992 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:33,993 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:54:33,995 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:54:35,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66f07d31]) (logid:d92369bf) No inactive management server node found 2025-06-29 02:54:35,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66f07d31]) (logid:d92369bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:36,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0316f337]) (logid:3ea3c395) No inactive management server node found 2025-06-29 02:54:36,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0316f337]) (logid:3ea3c395) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:37,694 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6ede1545]) (logid:aa95d0de) HA health check task is running... 2025-06-29 02:54:38,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39739475]) (logid:b6caf891) No inactive management server node found 2025-06-29 02:54:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39739475]) (logid:b6caf891) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:39,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-797ac4a2]) (logid:df78d7c7) No inactive management server node found 2025-06-29 02:54:39,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-797ac4a2]) (logid:df78d7c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:40,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7934: Seq 7-7934: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:54:40,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7934 2025-06-29 02:54:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b76db7d0]) (logid:f0dd70a5) Found 0 routers to update status. 2025-06-29 02:54:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b76db7d0]) (logid:f0dd70a5) Found 0 VPC's to update Redundant State. 2025-06-29 02:54:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b76db7d0]) (logid:f0dd70a5) Found 0 networks to update RvR status. 2025-06-29 02:54:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-b5c337aa]) (logid:05bb129d) DbCollector is running... 2025-06-29 02:54:41,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-145e6a69]) (logid:6cc51f27) No inactive management server node found 2025-06-29 02:54:41,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-145e6a69]) (logid:6cc51f27) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:41,695 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7fa6fb7c]) (logid:d1ec0b16) HA health check task is running... 2025-06-29 02:54:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) ManagementServerCollector is running... 2025-06-29 02:54:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) DB properties were already loaded 2025-06-29 02:54:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:54:42,169 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Execution of process [4062319] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:54:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Exit value of process [4062319] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:54:42,169 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Process [4062319] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:54:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) usage install: null 2025-06-29 02:54:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Sessions active: 0 2025-06-29 02:54:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:54:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062322] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:54:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:54:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062327] for command [/bin/bash -c uname -r ]. 2025-06-29 02:54:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:54:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062328] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:54:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) system memory from /proc: 4887515136 2025-06-29 02:54:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:54:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062333] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:54:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) free memory from /proc: 502202368 2025-06-29 02:54:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:54:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062338] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:54:42,210 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) used memory from /proc: 761716 2025-06-29 02:54:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:54:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062339] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:54:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:54:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062340] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:54:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:54:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062341] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:54:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:54:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062348] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:54:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:54:42,250 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Successfully executed process [4062349] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:54:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:54:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail buffersdirect.count, 120 2025-06-29 02:54:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail buffersdirect.used, 657986 2025-06-29 02:54:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:54:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail buffersmapped.count, 7 2025-06-29 02:54:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail buffersmapped.used, 798602 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail gcPS-Scavenge.count, 488 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail gcPS-Scavenge.time, 3568 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail jvmuptime, 42422330 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memoryheap.committed, 351272960 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memoryheap.init, 77594624 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memoryheap.usage, 0.10260383168015956 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorynon-heap.max, -1 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorynon-heap.usage, -2.20139312E8 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorynon-heap.used, 220139312 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17566877999200214 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30044968673976075 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.Metaspace.usage, 0.9855652260032314 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.PS-Eden-Space.usage, 0.029540025243219338 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.PS-Old-Gen.usage, 0.11960459052316001 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorypools.PS-Survivor-Space.usage, 0.39375 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorytotal.committed, 576192512 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorytotal.init, 85262336 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorytotal.max, 1908932607 2025-06-29 02:54:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail memorytotal.used, 416059360 2025-06-29 02:54:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail threadsdeadlocks, [] 2025-06-29 02:54:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail threadsnew.count, 0 2025-06-29 02:54:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:54:42,257 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-f01b5baf]) (logid:9f7dd3e5) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:54:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":156179920,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42422246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":415232352,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195863800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":502202368,"systemMemoryUsed":761716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.14,0.19],"systemCyclesUsage":[66364798,53805418,4587689878],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:54:42,257 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1413, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:54:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":156179920,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42422246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":415232352,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195863800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":502202368,"systemMemoryUsed":761716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.14,0.19],"systemCyclesUsage":[66364798,53805418,4587689878],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:54:42,257 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:54:42,261 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-317:[ctx-7ed006e8]) (logid:d81e7058) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:54:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":156179920,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42422246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":415232352,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195863800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":502202368,"systemMemoryUsed":761716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.14,0.19],"systemCyclesUsage":[66364798,53805418,4587689878],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:54:42,261 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:54:42,261 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1413, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:54:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351272960,"freeJvmMemoryBytes":156179920,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42422246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":415232352,"totalCommitted":576192512,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195863800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":502202368,"systemMemoryUsed":761716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.14,0.19],"systemCyclesUsage":[66364798,53805418,4587689878],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:54:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07b521dc]) (logid:757b18ab) No inactive management server node found 2025-06-29 02:54:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07b521dc]) (logid:757b18ab) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5ad8f09]) (logid:15a93ef4) No inactive management server node found 2025-06-29 02:54:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5ad8f09]) (logid:15a93ef4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:54:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:54:45,696 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-eaaf19ef]) (logid:ca544d91) HA health check task is running... 2025-06-29 02:54:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0e57ec4]) (logid:19bbdbc9) No inactive management server node found 2025-06-29 02:54:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0e57ec4]) (logid:19bbdbc9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3d0cb35a]) (logid:b59276c9) VM scheduler.poll is being called at 2025-06-29T00:55:00+0000 2025-06-29 02:54:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3d0cb35a]) (logid:b59276c9) Got 0 scheduled jobs to be executed at 2025-06-29T00:55:00+0000 2025-06-29 02:54:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3d0cb35a]) (logid:b59276c9) Cleaned up 0 VM scheduled job entries 2025-06-29 02:54:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-629b8c08]) (logid:db3ab339) VM Operation Thread Running 2025-06-29 02:54:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-97539e20]) (logid:4d669576) ClusterDRS.poll is being called at 2025-06-29T00:55:00+0000 2025-06-29 02:54:46,337 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-97539e20]) (logid:4d669576) Removed 0 old drs migration plans 2025-06-29 02:54:47,232 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:47,233 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:47,234 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:54:47,237 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:54:47,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6866d16d]) (logid:a5bfea63) No inactive management server node found 2025-06-29 02:54:47,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6866d16d]) (logid:a5bfea63) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8d731f2]) (logid:8bc6945f) No inactive management server node found 2025-06-29 02:54:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8d731f2]) (logid:8bc6945f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:49,698 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-60a1cc6e]) (logid:0f22abe2) HA health check task is running... 2025-06-29 02:54:50,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7936: Seq 7-7936: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:54:50,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7936 2025-06-29 02:54:50,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d36696b0]) (logid:03101d3c) No inactive management server node found 2025-06-29 02:54:50,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d36696b0]) (logid:03101d3c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-704fa3a2]) (logid:f92f800d) No inactive management server node found 2025-06-29 02:54:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-704fa3a2]) (logid:f92f800d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ababf0b]) (logid:6a74e0f4) No inactive management server node found 2025-06-29 02:54:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ababf0b]) (logid:6a74e0f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:53,699 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c506dd6a]) (logid:35f54f2d) HA health check task is running... 2025-06-29 02:54:54,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-191b1256]) (logid:ff1fae2f) No inactive management server node found 2025-06-29 02:54:54,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-191b1256]) (logid:ff1fae2f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:56,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-117aad65]) (logid:d2a4ba9c) No inactive management server node found 2025-06-29 02:54:56,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-117aad65]) (logid:d2a4ba9c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:54:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-69ec9694]) (logid:eb15db55) Enabled non-edge zones available for scan: 1 2025-06-29 02:54:56,764 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-69ec9694]) (logid:eb15db55) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:54:56,766 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-69ec9694]) (logid:eb15db55) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:54:56,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a7944a6f]) (logid:c42e7a7b) Enabled non-edge zones available for scan: 1 2025-06-29 02:54:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a7944a6f]) (logid:c42e7a7b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:54:57,700 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-98faabb4]) (logid:cf97a487) HA health check task is running... 2025-06-29 02:54:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc1b536c]) (logid:cae1c0eb) No inactive management server node found 2025-06-29 02:54:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc1b536c]) (logid:cae1c0eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:57,885 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-2b3e563d]) (logid:c5d241d2) AutoScaling Monitor is running... 2025-06-29 02:54:57,996 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-eaa02b2c]) (logid:d8876bc6) Starting to remove all volume_stats rows older than [Sat Jun 28 14:54:57 CEST 2025]. 2025-06-29 02:54:57,996 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 02:54:57,998 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-eaa02b2c]) (logid:d8876bc6) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:54:57 CEST 2025]. 2025-06-29 02:54:58,597 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) StorageCollector is running... 2025-06-29 02:54:58,602 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:54:58,602 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:54:58,602 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:54:58,603 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-6717118844223095471: Routed from 90520738888109 2025-06-29 02:54:58,656 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 4-6717118844223095471: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:54:58,660 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:54:58,660 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:54:58,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:54:58,661 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-3048092522799696651: Routed from 90520738888109 2025-06-29 02:54:58,708 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 1-3048092522799696651: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:54:58,730 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:54:58,730 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:54:58,730 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:54:58,731 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-1612007191621929926: Routed from 90520738888109 2025-06-29 02:54:58,781 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-1612007191621929926: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:54:58,793 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:54:58,793 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:54:58,793 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:54:58,794 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-3048092522799696652: Routed from 90520738888109 2025-06-29 02:54:59,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2041c49f]) (logid:2991f075) No inactive management server node found 2025-06-29 02:54:59,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2041c49f]) (logid:2991f075) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:54:59,846 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 1-3048092522799696652: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:54:59,848 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:54:59,848 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:54:59,848 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:54:59,849 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-1612007191621929927: Routed from 90520738888109 2025-06-29 02:55:00,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7937: Seq 7-7937: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:55:00,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7937 2025-06-29 02:55:00,244 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:55:00,248 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:55:00,250 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) Seq 2-3048092522799696653: Routed from 90520738888109 2025-06-29 02:55:00,262 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) Seq 1-3048092522799696653: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:55:00,296 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:55:00,297 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) Seq 2-1612007191621929928: Routed from 90520738888109 2025-06-29 02:55:00,353 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-3ffb4cf6]) (logid:f1bbeb0f) Seq 2-1612007191621929928: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:55:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2119cb20]) (logid:c21d196d) No inactive management server node found 2025-06-29 02:55:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2119cb20]) (logid:c21d196d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:00,910 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6a61145e]) (logid:a96625a3) Seq 2-1612007191621929927: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:55:01,701 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8ecbbd18]) (logid:c297b14d) HA health check task is running... 2025-06-29 02:55:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c727a9a7]) (logid:ce54013c) No inactive management server node found 2025-06-29 02:55:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c727a9a7]) (logid:ce54013c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:03,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e25c93f]) (logid:8096603e) No inactive management server node found 2025-06-29 02:55:03,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e25c93f]) (logid:8096603e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:05,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf838612]) (logid:b4e1bc61) No inactive management server node found 2025-06-29 02:55:05,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf838612]) (logid:b4e1bc61) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:05,703 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a1f97bbb]) (logid:8cd86c90) HA health check task is running... 2025-06-29 02:55:06,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-052ad957]) (logid:9ddaba9b) No inactive management server node found 2025-06-29 02:55:06,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-052ad957]) (logid:9ddaba9b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:08,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-470d36c1]) (logid:b2743c48) No inactive management server node found 2025-06-29 02:55:08,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-470d36c1]) (logid:b2743c48) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:09,704 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3f785ddc]) (logid:eac51bb2) HA health check task is running... 2025-06-29 02:55:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cd3744f]) (logid:cc94c774) No inactive management server node found 2025-06-29 02:55:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cd3744f]) (logid:cc94c774) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:10,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7938: Seq 7-7938: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:55:10,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7938 2025-06-29 02:55:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dd3c7a48]) (logid:c3510ec2) Found 0 routers to update status. 2025-06-29 02:55:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dd3c7a48]) (logid:c3510ec2) Found 0 VPC's to update Redundant State. 2025-06-29 02:55:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dd3c7a48]) (logid:c3510ec2) Found 0 networks to update RvR status. 2025-06-29 02:55:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b112784]) (logid:16d32535) No inactive management server node found 2025-06-29 02:55:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b112784]) (logid:16d32535) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eac0a991]) (logid:7c87c97a) No inactive management server node found 2025-06-29 02:55:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eac0a991]) (logid:7c87c97a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:13,139 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-b005b0da]) (logid:db1d9c63) Starting to remove all vm_stats rows older than [Sat Jun 28 14:55:13 CEST 2025]. 2025-06-29 02:55:13,168 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-b005b0da]) (logid:db1d9c63) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:55:13 CEST 2025]. 2025-06-29 02:55:13,705 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b3ea49fc]) (logid:a4893908) HA health check task is running... 2025-06-29 02:55:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-661ff00d]) (logid:ed7d05ac) No inactive management server node found 2025-06-29 02:55:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-661ff00d]) (logid:ed7d05ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8818d9ce]) (logid:9d4628c3) No inactive management server node found 2025-06-29 02:55:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8818d9ce]) (logid:9d4628c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-13ebcd8f]) (logid:c24c7315) Resetting hosts suitable for reconnect 2025-06-29 02:55:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-13ebcd8f]) (logid:c24c7315) Completed resetting hosts suitable for reconnect 2025-06-29 02:55:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-13ebcd8f]) (logid:c24c7315) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:55:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-13ebcd8f]) (logid:c24c7315) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:55:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-13ebcd8f]) (logid:c24c7315) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:55:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-13ebcd8f]) (logid:c24c7315) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:55:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c75e3651]) (logid:6477511e) No inactive management server node found 2025-06-29 02:55:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c75e3651]) (logid:6477511e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:55:17,707 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-761bcbf6]) (logid:d88408d6) HA health check task is running... 2025-06-29 02:55:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd6b5968]) (logid:13e6ac65) No inactive management server node found 2025-06-29 02:55:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd6b5968]) (logid:13e6ac65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:20,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-7939: Seq 7-7939: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:55:20,108 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 7939 2025-06-29 02:55:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-384cd975]) (logid:bfba403f) No inactive management server node found 2025-06-29 02:55:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-384cd975]) (logid:bfba403f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:21,708 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3fb67b2c]) (logid:65775108) HA health check task is running... 2025-06-29 02:55:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d97dccd]) (logid:50fe56fc) No inactive management server node found 2025-06-29 02:55:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d97dccd]) (logid:50fe56fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:23,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-228f1336]) (logid:bd60ec8a) No inactive management server node found 2025-06-29 02:55:23,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-228f1336]) (logid:bd60ec8a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:24,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-216a4c49]) (logid:67a19480) No inactive management server node found 2025-06-29 02:55:24,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-216a4c49]) (logid:67a19480) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:25,709 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9da15db1]) (logid:667cd139) HA health check task is running... 2025-06-29 02:55:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9247f93]) (logid:bcad471b) No inactive management server node found 2025-06-29 02:55:26,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9247f93]) (logid:bcad471b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-03ab87c6]) (logid:ef7a957d) Enabled non-edge zones available for scan: 1 2025-06-29 02:55:26,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-03ab87c6]) (logid:ef7a957d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:55:26,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-03ab87c6]) (logid:ef7a957d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:55:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-80256e18]) (logid:87f95e0e) Enabled non-edge zones available for scan: 1 2025-06-29 02:55:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-80256e18]) (logid:87f95e0e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:55:27,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2bff78d]) (logid:b11be3da) No inactive management server node found 2025-06-29 02:55:27,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2bff78d]) (logid:b11be3da) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:29,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76662570]) (logid:962dda13) No inactive management server node found 2025-06-29 02:55:29,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76662570]) (logid:962dda13) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:55:29,710 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c0bbe793]) (logid:4b66f1be) HA health check task is running... 2025-06-29 02:55:29,908 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:55:29,909 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:55:29,910 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) Seq 2-3048092522799696654: Routed from 90520738888109 2025-06-29 02:55:29,952 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) Seq 1-3048092522799696654: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:55:29,953 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:55:29,954 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) Seq 2-1612007191621929929: Routed from 90520738888109 2025-06-29 02:55:29,956 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dda76096]) (logid:4aa42617) Seq 2-1612007191621929929: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:55:30,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7940: Seq 7-7940: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:55:30,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7940 2025-06-29 02:55:30,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75db9c32]) (logid:9562a794) No inactive management server node found 2025-06-29 02:55:30,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75db9c32]) (logid:9562a794) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:32,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0719f677]) (logid:a1db6873) No inactive management server node found 2025-06-29 02:55:32,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0719f677]) (logid:a1db6873) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:33,712 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-95e641ee]) (logid:eafe9307) HA health check task is running... 2025-06-29 02:55:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e09b2f91]) (logid:54001295) No inactive management server node found 2025-06-29 02:55:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e09b2f91]) (logid:54001295) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:55:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:55:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:55:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:55:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea3fbc64]) (logid:e2ac24a3) No inactive management server node found 2025-06-29 02:55:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea3fbc64]) (logid:e2ac24a3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08effec1]) (logid:1181208e) No inactive management server node found 2025-06-29 02:55:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08effec1]) (logid:1181208e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:37,713 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-dd2a38be]) (logid:294d14ff) HA health check task is running... 2025-06-29 02:55:38,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac150af7]) (logid:7d1d630e) No inactive management server node found 2025-06-29 02:55:38,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac150af7]) (logid:7d1d630e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9acb1ca4]) (logid:432e5554) No inactive management server node found 2025-06-29 02:55:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9acb1ca4]) (logid:432e5554) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:40,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7941: Seq 7-7941: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:55:40,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7941 2025-06-29 02:55:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-571d7b3f]) (logid:a9dba592) Found 0 routers to update status. 2025-06-29 02:55:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-571d7b3f]) (logid:a9dba592) Found 0 VPC's to update Redundant State. 2025-06-29 02:55:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-571d7b3f]) (logid:a9dba592) Found 0 networks to update RvR status. 2025-06-29 02:55:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-0efc64d3]) (logid:5e574269) DbCollector is running... 2025-06-29 02:55:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f622ac4]) (logid:c8921a3d) No inactive management server node found 2025-06-29 02:55:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f622ac4]) (logid:c8921a3d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:41,715 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6a6f2957]) (logid:2977fed5) HA health check task is running... 2025-06-29 02:55:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) ManagementServerCollector is running... 2025-06-29 02:55:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) DB properties were already loaded 2025-06-29 02:55:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:55:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Execution of process [4062356] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:55:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Exit value of process [4062356] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:55:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Process [4062356] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:55:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) usage install: null 2025-06-29 02:55:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Sessions active: 0 2025-06-29 02:55:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:55:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062359] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:55:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:55:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062364] for command [/bin/bash -c uname -r ]. 2025-06-29 02:55:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:55:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062365] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:55:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) system memory from /proc: 4887515136 2025-06-29 02:55:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:55:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062370] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:55:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) free memory from /proc: 500924416 2025-06-29 02:55:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:55:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062375] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:55:42,204 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) used memory from /proc: 761692 2025-06-29 02:55:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:55:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062376] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:55:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:55:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062377] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:55:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:55:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062378] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:55:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:55:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062385] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:55:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:55:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Successfully executed process [4062386] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail buffersdirect.count, 120 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail buffersdirect.used, 657986 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail buffersmapped.count, 7 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail buffersmapped.used, 798602 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail gcPS-Scavenge.count, 489 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail gcPS-Scavenge.time, 3571 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail jvmuptime, 42482319 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memoryheap.committed, 351797248 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memoryheap.init, 77594624 2025-06-29 02:55:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memoryheap.usage, 0.10049884380203326 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorynon-heap.max, -1 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorynon-heap.usage, -2.20142368E8 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorynon-heap.used, 220142368 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17566877999200214 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3004673909421135 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.Metaspace.usage, 0.9855711596021598 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.PS-Eden-Space.usage, 0.02365420935288915 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.PS-Old-Gen.usage, 0.11966752534556939 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorypools.PS-Survivor-Space.usage, 0.39218902587890625 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorytotal.committed, 576716800 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorytotal.init, 85262336 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorytotal.max, 1908932607 2025-06-29 02:55:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail memorytotal.used, 412030528 2025-06-29 02:55:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail threadsdeadlocks, [] 2025-06-29 02:55:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail threadsnew.count, 0 2025-06-29 02:55:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:55:42,246 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-7f07db70]) (logid:67082bbe) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:55:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":160697856,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42482246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":411241760,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191845520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500924416,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.14,0.18],"systemCyclesUsage":[66365121,53805737,4587712968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:55:42,247 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1415, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:55:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":160697856,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42482246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":411241760,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191845520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500924416,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.14,0.18],"systemCyclesUsage":[66365121,53805737,4587712968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:55:42,247 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:55:42,250 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-317:[ctx-18a197ff]) (logid:266220e0) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:55:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":160697856,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42482246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":411241760,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191845520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500924416,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.14,0.18],"systemCyclesUsage":[66365121,53805737,4587712968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:55:42,251 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 02:55:42,251 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1415, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:55:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":160697856,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42482246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":411241760,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191845520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500924416,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.14,0.18],"systemCyclesUsage":[66365121,53805737,4587712968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:55:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e930fefc]) (logid:bcbc0f62) No inactive management server node found 2025-06-29 02:55:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e930fefc]) (logid:bcbc0f62) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f00685ab]) (logid:9826f190) No inactive management server node found 2025-06-29 02:55:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f00685ab]) (logid:9826f190) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:55:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:55:45,716 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5a678e76]) (logid:1089baa7) HA health check task is running... 2025-06-29 02:55:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-768b5c95]) (logid:57047899) No inactive management server node found 2025-06-29 02:55:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-768b5c95]) (logid:57047899) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-caee1b85]) (logid:10767866) VM scheduler.poll is being called at 2025-06-29T00:56:00+0000 2025-06-29 02:55:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-caee1b85]) (logid:10767866) Got 0 scheduled jobs to be executed at 2025-06-29T00:56:00+0000 2025-06-29 02:55:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-caee1b85]) (logid:10767866) Cleaned up 0 VM scheduled job entries 2025-06-29 02:55:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-697f711e]) (logid:3b2dac03) VM Operation Thread Running 2025-06-29 02:55:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ac79bece]) (logid:47e27079) ClusterDRS.poll is being called at 2025-06-29T00:56:00+0000 2025-06-29 02:55:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ac79bece]) (logid:47e27079) Removed 0 old drs migration plans 2025-06-29 02:55:47,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:55:47,233 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:55:47,234 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:55:47,236 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:55:47,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9b3cc99]) (logid:87c785c8) No inactive management server node found 2025-06-29 02:55:47,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9b3cc99]) (logid:87c785c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:48,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79720f48]) (logid:f19186c7) No inactive management server node found 2025-06-29 02:55:48,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79720f48]) (logid:f19186c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:49,717 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-30bfcd03]) (logid:1b9c4f91) HA health check task is running... 2025-06-29 02:55:50,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7943: Seq 7-7943: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:55:50,099 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7943 2025-06-29 02:55:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9d3845a]) (logid:4317a69b) No inactive management server node found 2025-06-29 02:55:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9d3845a]) (logid:4317a69b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:51,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9db3cec3]) (logid:74153bef) No inactive management server node found 2025-06-29 02:55:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9db3cec3]) (logid:74153bef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-294d4caa]) (logid:4ad91f5c) No inactive management server node found 2025-06-29 02:55:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-294d4caa]) (logid:4ad91f5c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:55:53,718 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-84ad6d33]) (logid:f33a39a2) HA health check task is running... 2025-06-29 02:55:54,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b88234d6]) (logid:5eaf5b6a) No inactive management server node found 2025-06-29 02:55:54,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b88234d6]) (logid:5eaf5b6a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73b31ee5]) (logid:0cf674e4) No inactive management server node found 2025-06-29 02:55:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73b31ee5]) (logid:0cf674e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-037b4e06]) (logid:57abe150) Enabled non-edge zones available for scan: 1 2025-06-29 02:55:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-037b4e06]) (logid:57abe150) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:55:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-037b4e06]) (logid:57abe150) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:55:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7388a934]) (logid:6db0493b) Enabled non-edge zones available for scan: 1 2025-06-29 02:55:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7388a934]) (logid:6db0493b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:55:57,719 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5964853f]) (logid:44cd1947) HA health check task is running... 2025-06-29 02:55:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94c41aaa]) (logid:e0582db4) No inactive management server node found 2025-06-29 02:55:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94c41aaa]) (logid:e0582db4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:55:57,887 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-8a55904e]) (logid:a2ca1d72) AutoScaling Monitor is running... 2025-06-29 02:55:58,001 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-617a34ae]) (logid:252a583a) Starting to remove all volume_stats rows older than [Sat Jun 28 14:55:57 CEST 2025]. 2025-06-29 02:55:58,002 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-617a34ae]) (logid:252a583a) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:55:57 CEST 2025]. 2025-06-29 02:55:59,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9df782c5]) (logid:62c51405) No inactive management server node found 2025-06-29 02:55:59,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9df782c5]) (logid:62c51405) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:00,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7944: Seq 7-7944: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:00,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7944 2025-06-29 02:56:00,407 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:56:00,410 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:56:00,411 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) Seq 2-3048092522799696655: Routed from 90520738888109 2025-06-29 02:56:00,464 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) Seq 1-3048092522799696655: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:56:00,511 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:56:00,512 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) Seq 2-1612007191621929930: Routed from 90520738888109 2025-06-29 02:56:00,558 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c11adedb]) (logid:bbaf5261) Seq 2-1612007191621929930: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:56:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8dd1afe]) (logid:6d3ce4d1) No inactive management server node found 2025-06-29 02:56:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8dd1afe]) (logid:6d3ce4d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:00,910 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) StorageCollector is running... 2025-06-29 02:56:00,914 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:56:00,914 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:56:00,914 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:56:00,915 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-6717118844223095472: Routed from 90520738888109 2025-06-29 02:56:00,969 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 4-6717118844223095472: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:56:00,972 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:56:00,972 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:56:00,972 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:56:00,973 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-3048092522799696656: Routed from 90520738888109 2025-06-29 02:56:01,038 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 1-3048092522799696656: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:56:01,048 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:56:01,048 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:56:01,048 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:56:01,049 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-1612007191621929931: Routed from 90520738888109 2025-06-29 02:56:01,059 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-1612007191621929931: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:56:01,069 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:56:01,069 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:56:01,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:56:01,069 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-3048092522799696657: Routed from 90520738888109 2025-06-29 02:56:01,721 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c698a8be]) (logid:91b17d71) HA health check task is running... 2025-06-29 02:56:02,161 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 1-3048092522799696657: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:56:02,163 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:56:02,163 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:56:02,163 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:56:02,164 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-1612007191621929932: Routed from 90520738888109 2025-06-29 02:56:02,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1bd5c08]) (logid:7455511a) No inactive management server node found 2025-06-29 02:56:02,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1bd5c08]) (logid:7455511a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:03,227 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-785bf328]) (logid:38207ff9) Seq 2-1612007191621929932: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:56:03,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5577093]) (logid:3a2b2db3) No inactive management server node found 2025-06-29 02:56:03,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5577093]) (logid:3a2b2db3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1459026]) (logid:bd1ee4f0) No inactive management server node found 2025-06-29 02:56:05,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1459026]) (logid:bd1ee4f0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:05,722 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7823a6d7]) (logid:6b0de026) HA health check task is running... 2025-06-29 02:56:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0e1d9a4]) (logid:f08feb7a) No inactive management server node found 2025-06-29 02:56:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0e1d9a4]) (logid:f08feb7a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5fd93e7]) (logid:e11f8b6d) No inactive management server node found 2025-06-29 02:56:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5fd93e7]) (logid:e11f8b6d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:09,723 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4abb3e47]) (logid:d33965ff) HA health check task is running... 2025-06-29 02:56:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-047acc31]) (logid:0e260448) No inactive management server node found 2025-06-29 02:56:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-047acc31]) (logid:0e260448) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:10,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7945: Seq 7-7945: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:10,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7945 2025-06-29 02:56:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d9438878]) (logid:733c4d74) Found 0 routers to update status. 2025-06-29 02:56:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d9438878]) (logid:733c4d74) Found 0 VPC's to update Redundant State. 2025-06-29 02:56:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d9438878]) (logid:733c4d74) Found 0 networks to update RvR status. 2025-06-29 02:56:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-525624ac]) (logid:5eed78e9) No inactive management server node found 2025-06-29 02:56:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-525624ac]) (logid:5eed78e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28708d65]) (logid:e99ce65f) No inactive management server node found 2025-06-29 02:56:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28708d65]) (logid:e99ce65f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:13,170 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-ccedb3ed]) (logid:1da164e0) Starting to remove all vm_stats rows older than [Sat Jun 28 14:56:13 CEST 2025]. 2025-06-29 02:56:13,193 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-ccedb3ed]) (logid:1da164e0) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:56:13 CEST 2025]. 2025-06-29 02:56:13,724 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a1518ac6]) (logid:52cfd269) HA health check task is running... 2025-06-29 02:56:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ecca19e]) (logid:05c4dba4) No inactive management server node found 2025-06-29 02:56:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ecca19e]) (logid:05c4dba4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe75fb18]) (logid:3644bcff) No inactive management server node found 2025-06-29 02:56:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe75fb18]) (logid:3644bcff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80414a6a]) (logid:08c6ce25) No inactive management server node found 2025-06-29 02:56:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80414a6a]) (logid:08c6ce25) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:17,726 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e1fdde0d]) (logid:f7d27f27) HA health check task is running... 2025-06-29 02:56:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2231918f]) (logid:7f4b82a4) No inactive management server node found 2025-06-29 02:56:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2231918f]) (logid:7f4b82a4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:20,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7946: Seq 7-7946: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:20,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7946 2025-06-29 02:56:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2478b18f]) (logid:7e3cc17c) No inactive management server node found 2025-06-29 02:56:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2478b18f]) (logid:7e3cc17c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:21,727 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-cabe4f23]) (logid:259d3cac) HA health check task is running... 2025-06-29 02:56:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52eae76a]) (logid:5020f1d0) No inactive management server node found 2025-06-29 02:56:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52eae76a]) (logid:5020f1d0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:23,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcf75610]) (logid:1d03099a) No inactive management server node found 2025-06-29 02:56:23,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcf75610]) (logid:1d03099a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:24,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2b86f08]) (logid:10af90ec) No inactive management server node found 2025-06-29 02:56:24,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2b86f08]) (logid:10af90ec) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:25,728 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-fbdbbccc]) (logid:cfee2be8) HA health check task is running... 2025-06-29 02:56:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91c74a13]) (logid:cf8a8336) No inactive management server node found 2025-06-29 02:56:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91c74a13]) (logid:cf8a8336) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e18e754a]) (logid:c211243f) Enabled non-edge zones available for scan: 1 2025-06-29 02:56:26,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e18e754a]) (logid:c211243f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:56:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e18e754a]) (logid:c211243f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:56:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0982f854]) (logid:82245157) Enabled non-edge zones available for scan: 1 2025-06-29 02:56:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0982f854]) (logid:82245157) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:56:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e27be38]) (logid:29dbe1a1) No inactive management server node found 2025-06-29 02:56:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e27be38]) (logid:29dbe1a1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35eedcca]) (logid:e885bcb5) No inactive management server node found 2025-06-29 02:56:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35eedcca]) (logid:e885bcb5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:29,730 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-38b01bf9]) (logid:51ad394d) HA health check task is running... 2025-06-29 02:56:29,959 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:56:29,959 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:56:29,960 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) Seq 2-3048092522799696658: Routed from 90520738888109 2025-06-29 02:56:29,962 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) Seq 1-3048092522799696658: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:56:29,963 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:56:29,964 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) Seq 2-1612007191621929933: Routed from 90520738888109 2025-06-29 02:56:29,967 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-48b91c72]) (logid:8d7c5bde) Seq 2-1612007191621929933: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:56:30,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7947: Seq 7-7947: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:30,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7947 2025-06-29 02:56:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d68a3918]) (logid:835731b5) No inactive management server node found 2025-06-29 02:56:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d68a3918]) (logid:835731b5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:32,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f624e86c]) (logid:4b4da622) No inactive management server node found 2025-06-29 02:56:32,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f624e86c]) (logid:4b4da622) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:56:33,731 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0ccbd204]) (logid:61fd4794) HA health check task is running... 2025-06-29 02:56:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e95f497a]) (logid:53db2d68) No inactive management server node found 2025-06-29 02:56:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e95f497a]) (logid:53db2d68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:56:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:56:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:56:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:56:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfe526f9]) (logid:90482cfe) No inactive management server node found 2025-06-29 02:56:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfe526f9]) (logid:90482cfe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06a86c46]) (logid:511f198f) No inactive management server node found 2025-06-29 02:56:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06a86c46]) (logid:511f198f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:37,732 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d2883269]) (logid:3008ee40) HA health check task is running... 2025-06-29 02:56:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c82fbb1]) (logid:b6431dd5) No inactive management server node found 2025-06-29 02:56:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c82fbb1]) (logid:b6431dd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e56fb3a]) (logid:e98d156c) No inactive management server node found 2025-06-29 02:56:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e56fb3a]) (logid:e98d156c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:40,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7948: Seq 7-7948: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:40,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7948 2025-06-29 02:56:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-33db5e0d]) (logid:08a00e0a) Found 0 routers to update status. 2025-06-29 02:56:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-33db5e0d]) (logid:08a00e0a) Found 0 VPC's to update Redundant State. 2025-06-29 02:56:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-33db5e0d]) (logid:08a00e0a) Found 0 networks to update RvR status. 2025-06-29 02:56:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-ff97ca3e]) (logid:00d2057f) DbCollector is running... 2025-06-29 02:56:41,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ea6f479]) (logid:1c29edb3) No inactive management server node found 2025-06-29 02:56:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ea6f479]) (logid:1c29edb3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:41,734 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-87947a95]) (logid:43d5ba3a) HA health check task is running... 2025-06-29 02:56:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) ManagementServerCollector is running... 2025-06-29 02:56:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) DB properties were already loaded 2025-06-29 02:56:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:56:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Execution of process [4062401] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:56:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Exit value of process [4062401] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:56:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Process [4062401] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:56:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) usage install: null 2025-06-29 02:56:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Sessions active: 0 2025-06-29 02:56:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:56:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062404] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:56:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:56:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062409] for command [/bin/bash -c uname -r ]. 2025-06-29 02:56:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:56:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062410] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:56:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) system memory from /proc: 4887515136 2025-06-29 02:56:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:56:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062415] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:56:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) free memory from /proc: 510488576 2025-06-29 02:56:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:56:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062420] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:56:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) used memory from /proc: 761564 2025-06-29 02:56:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:56:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062421] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:56:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:56:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062422] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:56:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:56:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062423] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:56:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:56:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062430] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:56:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:56:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Successfully executed process [4062431] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail buffersdirect.count, 120 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail buffersdirect.used, 657986 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail buffersmapped.count, 7 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail buffersmapped.used, 798602 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail gcPS-Scavenge.count, 490 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail gcPS-Scavenge.time, 3575 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail jvmuptime, 42542321 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memoryheap.committed, 351797248 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memoryheap.init, 77594624 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memoryheap.usage, 0.09792137827004943 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorynon-heap.max, -1 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorynon-heap.usage, -2.20163744E8 2025-06-29 02:56:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorynon-heap.used, 220163744 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17584269194881366 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3004673909421135 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.Metaspace.usage, 0.9855711596021598 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.PS-Eden-Space.usage, 0.016530013174380895 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.PS-Old-Gen.usage, 0.11968468938804468 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorypools.PS-Survivor-Space.usage, 0.39218902587890625 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorytotal.committed, 576716800 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorytotal.init, 85262336 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorytotal.max, 1908932607 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail memorytotal.used, 407140224 2025-06-29 02:56:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail threadsdeadlocks, [] 2025-06-29 02:56:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail threadsnew.count, 0 2025-06-29 02:56:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:56:42,249 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-928b6319]) (logid:751e2b05) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:56:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":165629272,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42542245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":406331720,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":186925312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510488576,"systemMemoryUsed":761564,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.17],"systemCyclesUsage":[66365448,53806045,4587736083],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:56:42,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1417, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:56:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":165629272,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42542245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":406331720,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":186925312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510488576,"systemMemoryUsed":761564,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.17],"systemCyclesUsage":[66365448,53806045,4587736083],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:56:42,249 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:56:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 02:56:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1417, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:56:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":165629272,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42542245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":406331720,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":186925312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510488576,"systemMemoryUsed":761564,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.17],"systemCyclesUsage":[66365448,53806045,4587736083],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:56:42,255 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-318:[ctx-00309387]) (logid:57e1abd4) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:56:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":351797248,"freeJvmMemoryBytes":165629272,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42542245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":406331720,"totalCommitted":576716800,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":186925312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510488576,"systemMemoryUsed":761564,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.17],"systemCyclesUsage":[66365448,53806045,4587736083],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:56:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea2f7309]) (logid:5d8b23d0) No inactive management server node found 2025-06-29 02:56:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea2f7309]) (logid:5d8b23d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb43aa0b]) (logid:cc1ad642) No inactive management server node found 2025-06-29 02:56:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb43aa0b]) (logid:cc1ad642) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:45,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7949: Seq 7-7949: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:45,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7949 2025-06-29 02:56:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:56:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:56:45,735 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f63adf9e]) (logid:2b046b87) HA health check task is running... 2025-06-29 02:56:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cba0a4f1]) (logid:733202f3) No inactive management server node found 2025-06-29 02:56:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cba0a4f1]) (logid:733202f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-5824f930]) (logid:ed03aee3) VM scheduler.poll is being called at 2025-06-29T00:57:00+0000 2025-06-29 02:56:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-5824f930]) (logid:ed03aee3) Got 0 scheduled jobs to be executed at 2025-06-29T00:57:00+0000 2025-06-29 02:56:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-5824f930]) (logid:ed03aee3) Cleaned up 0 VM scheduled job entries 2025-06-29 02:56:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-5352a908]) (logid:59a60ae2) VM Operation Thread Running 2025-06-29 02:56:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a3a897bf]) (logid:2295329d) ClusterDRS.poll is being called at 2025-06-29T00:57:00+0000 2025-06-29 02:56:46,336 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a3a897bf]) (logid:2295329d) Removed 0 old drs migration plans 2025-06-29 02:56:47,052 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-54c11e08]) (logid:ea94862a) Resetting hosts suitable for reconnect 2025-06-29 02:56:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-54c11e08]) (logid:ea94862a) Completed resetting hosts suitable for reconnect 2025-06-29 02:56:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-54c11e08]) (logid:ea94862a) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:56:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-54c11e08]) (logid:ea94862a) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:56:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-54c11e08]) (logid:ea94862a) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:56:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-54c11e08]) (logid:ea94862a) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:56:47,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:56:47,267 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:56:47,268 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:56:47,271 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:56:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc9de6b4]) (logid:9e9970e7) No inactive management server node found 2025-06-29 02:56:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc9de6b4]) (logid:9e9970e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:48,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0b794f3]) (logid:4b43503e) No inactive management server node found 2025-06-29 02:56:48,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0b794f3]) (logid:4b43503e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:49,736 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1b01d150]) (logid:12964c10) HA health check task is running... 2025-06-29 02:56:50,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce15b2f2]) (logid:7a284dda) No inactive management server node found 2025-06-29 02:56:50,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce15b2f2]) (logid:7a284dda) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:51,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-921e9b2b]) (logid:99d56403) No inactive management server node found 2025-06-29 02:56:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-921e9b2b]) (logid:99d56403) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d756f58]) (logid:169d93f4) No inactive management server node found 2025-06-29 02:56:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d756f58]) (logid:169d93f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:53,738 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cb750c78]) (logid:ea75c7b9) HA health check task is running... 2025-06-29 02:56:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f4bf3ce]) (logid:7d66fbb0) No inactive management server node found 2025-06-29 02:56:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f4bf3ce]) (logid:7d66fbb0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:55,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7951: Seq 7-7951: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:56:55,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7951 2025-06-29 02:56:56,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-751e31ef]) (logid:924560af) No inactive management server node found 2025-06-29 02:56:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-751e31ef]) (logid:924560af) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7eedb3ce]) (logid:d601a488) Enabled non-edge zones available for scan: 1 2025-06-29 02:56:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-7eedb3ce]) (logid:d601a488) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:56:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7eedb3ce]) (logid:d601a488) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:56:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cf5e79a6]) (logid:7227d29e) Enabled non-edge zones available for scan: 1 2025-06-29 02:56:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cf5e79a6]) (logid:7227d29e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:56:57,739 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c9aa74f6]) (logid:9284772e) HA health check task is running... 2025-06-29 02:56:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89716b3c]) (logid:b2319a57) No inactive management server node found 2025-06-29 02:56:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89716b3c]) (logid:b2319a57) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:56:57,890 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-9fc8258a]) (logid:e575ff04) AutoScaling Monitor is running... 2025-06-29 02:56:58,005 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-7b16031b]) (logid:be2b49d4) Starting to remove all volume_stats rows older than [Sat Jun 28 14:56:58 CEST 2025]. 2025-06-29 02:56:58,006 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-7b16031b]) (logid:be2b49d4) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:56:58 CEST 2025]. 2025-06-29 02:56:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0bb4511]) (logid:059e4459) No inactive management server node found 2025-06-29 02:56:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0bb4511]) (logid:059e4459) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:00,611 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:57:00,614 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:57:00,615 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) Seq 2-3048092522799696659: Routed from 90520738888109 2025-06-29 02:57:00,668 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) Seq 1-3048092522799696659: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:57:00,714 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:57:00,714 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) Seq 2-1612007191621929934: Routed from 90520738888109 2025-06-29 02:57:00,760 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d0bb5f2e]) (logid:8784775c) Seq 2-1612007191621929934: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:57:00,819 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c5112d7]) (logid:95aa1237) No inactive management server node found 2025-06-29 02:57:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c5112d7]) (logid:95aa1237) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:01,740 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b8f53405]) (logid:ecd2562c) HA health check task is running... 2025-06-29 02:57:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36e64ca1]) (logid:a1bc7ab7) No inactive management server node found 2025-06-29 02:57:02,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36e64ca1]) (logid:a1bc7ab7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:03,227 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) StorageCollector is running... 2025-06-29 02:57:03,233 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:57:03,233 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:57:03,233 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:57:03,234 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 2-6717118844223095473: Routed from 90520738888109 2025-06-29 02:57:03,287 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 4-6717118844223095473: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:57:03,291 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:57:03,291 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:57:03,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:57:03,292 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 2-3048092522799696660: Routed from 90520738888109 2025-06-29 02:57:03,339 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 1-3048092522799696660: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:57:03,350 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:57:03,350 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:57:03,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:57:03,351 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 2-3048092522799696661: Routed from 90520738888109 2025-06-29 02:57:03,440 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 1-3048092522799696661: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:57:03,452 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:57:03,452 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:57:03,452 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:57:03,453 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 2-3048092522799696662: Routed from 90520738888109 2025-06-29 02:57:03,836 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7424af85]) (logid:a42dad66) No inactive management server node found 2025-06-29 02:57:03,836 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7424af85]) (logid:a42dad66) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:04,545 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 1-3048092522799696662: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:57:04,547 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:57:04,547 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:57:04,547 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:57:04,548 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 2-1612007191621929935: Routed from 90520738888109 2025-06-29 02:57:05,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7952: Seq 7-7952: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:57:05,108 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7952 2025-06-29 02:57:05,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5a89fa1]) (logid:2887bda0) No inactive management server node found 2025-06-29 02:57:05,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5a89fa1]) (logid:2887bda0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:05,566 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-52474323]) (logid:38fbf142) Seq 2-1612007191621929935: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:57:05,742 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d68ca8ce]) (logid:18f4fe56) HA health check task is running... 2025-06-29 02:57:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d3c5537]) (logid:a018275e) No inactive management server node found 2025-06-29 02:57:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d3c5537]) (logid:a018275e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0916936b]) (logid:d65cc95c) No inactive management server node found 2025-06-29 02:57:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0916936b]) (logid:d65cc95c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:09,743 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5535be5e]) (logid:e0e3cfb2) HA health check task is running... 2025-06-29 02:57:09,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7041b5b5]) (logid:004984ff) No inactive management server node found 2025-06-29 02:57:09,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7041b5b5]) (logid:004984ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-725b21ff]) (logid:bcf6f2bb) Found 0 routers to update status. 2025-06-29 02:57:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-725b21ff]) (logid:bcf6f2bb) Found 0 VPC's to update Redundant State. 2025-06-29 02:57:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-725b21ff]) (logid:bcf6f2bb) Found 0 networks to update RvR status. 2025-06-29 02:57:11,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25a630c6]) (logid:08bb67ee) No inactive management server node found 2025-06-29 02:57:11,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25a630c6]) (logid:08bb67ee) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b763ff3]) (logid:556455ee) No inactive management server node found 2025-06-29 02:57:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b763ff3]) (logid:556455ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:13,195 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-f2561e35]) (logid:123db440) Starting to remove all vm_stats rows older than [Sat Jun 28 14:57:13 CEST 2025]. 2025-06-29 02:57:13,218 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-f2561e35]) (logid:123db440) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:57:13 CEST 2025]. 2025-06-29 02:57:13,744 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7b26e0b9]) (logid:6b5ff48b) HA health check task is running... 2025-06-29 02:57:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc42e2d]) (logid:c7633eef) No inactive management server node found 2025-06-29 02:57:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc42e2d]) (logid:c7633eef) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:57:15,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7953: Seq 7-7953: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:57:15,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7953 2025-06-29 02:57:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6562cd12]) (logid:aaffdfb1) No inactive management server node found 2025-06-29 02:57:15,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6562cd12]) (logid:aaffdfb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24520198]) (logid:810f2e36) No inactive management server node found 2025-06-29 02:57:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24520198]) (logid:810f2e36) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:57:17,746 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5149c6ba]) (logid:3a92eb2f) HA health check task is running... 2025-06-29 02:57:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5118af70]) (logid:c76e9dd9) No inactive management server node found 2025-06-29 02:57:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5118af70]) (logid:c76e9dd9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:20,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-699dacc9]) (logid:668ef027) No inactive management server node found 2025-06-29 02:57:20,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-699dacc9]) (logid:668ef027) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:21,747 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5b7f3138]) (logid:70034a93) HA health check task is running... 2025-06-29 02:57:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86802615]) (logid:5afa206c) No inactive management server node found 2025-06-29 02:57:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86802615]) (logid:5afa206c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b848e0cc]) (logid:7ecbef5a) No inactive management server node found 2025-06-29 02:57:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b848e0cc]) (logid:7ecbef5a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86a47eea]) (logid:213787ac) No inactive management server node found 2025-06-29 02:57:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86a47eea]) (logid:213787ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:25,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-7954: Seq 7-7954: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:57:25,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 7954 2025-06-29 02:57:25,749 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3921aba4]) (logid:bc86f41d) HA health check task is running... 2025-06-29 02:57:26,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdb8372a]) (logid:772ea31a) No inactive management server node found 2025-06-29 02:57:26,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdb8372a]) (logid:772ea31a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-08b167c3]) (logid:c702c96c) Enabled non-edge zones available for scan: 1 2025-06-29 02:57:26,768 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-08b167c3]) (logid:c702c96c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:57:26,769 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-08b167c3]) (logid:c702c96c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:57:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0a9cb94f]) (logid:e83656a2) Enabled non-edge zones available for scan: 1 2025-06-29 02:57:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0a9cb94f]) (logid:e83656a2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:57:27,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b30ecb1e]) (logid:c9f630c6) No inactive management server node found 2025-06-29 02:57:27,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b30ecb1e]) (logid:c9f630c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:29,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-209960c9]) (logid:619c4a44) No inactive management server node found 2025-06-29 02:57:29,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-209960c9]) (logid:619c4a44) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:29,750 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bb0a6439]) (logid:cce3d52c) HA health check task is running... 2025-06-29 02:57:29,969 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:57:29,970 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:57:29,971 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) Seq 2-3048092522799696663: Routed from 90520738888109 2025-06-29 02:57:29,973 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) Seq 1-3048092522799696663: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:57:29,974 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:57:29,974 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) Seq 2-1612007191621929936: Routed from 90520738888109 2025-06-29 02:57:29,977 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-dcd43d42]) (logid:27c38a5a) Seq 2-1612007191621929936: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:57:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93228d2b]) (logid:a8704d20) No inactive management server node found 2025-06-29 02:57:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93228d2b]) (logid:a8704d20) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:32,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6ac8316]) (logid:0dcb10fd) No inactive management server node found 2025-06-29 02:57:32,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6ac8316]) (logid:0dcb10fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:33,751 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-30404125]) (logid:06648c67) HA health check task is running... 2025-06-29 02:57:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0817e4a]) (logid:70c900a8) No inactive management server node found 2025-06-29 02:57:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0817e4a]) (logid:70c900a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:33,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:57:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:57:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:57:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:57:35,094 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7955: Seq 7-7955: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:57:35,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7955 2025-06-29 02:57:35,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6366e858]) (logid:5bbf4992) No inactive management server node found 2025-06-29 02:57:35,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6366e858]) (logid:5bbf4992) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:57:36,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c306ebdd]) (logid:d8c9cf64) No inactive management server node found 2025-06-29 02:57:36,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c306ebdd]) (logid:d8c9cf64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:37,753 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-874327ca]) (logid:d698181f) HA health check task is running... 2025-06-29 02:57:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be467b0e]) (logid:e3154a9d) No inactive management server node found 2025-06-29 02:57:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be467b0e]) (logid:e3154a9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:39,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d29ffb4]) (logid:2753512a) No inactive management server node found 2025-06-29 02:57:39,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d29ffb4]) (logid:2753512a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85ec942c]) (logid:2629d9be) Found 0 routers to update status. 2025-06-29 02:57:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85ec942c]) (logid:2629d9be) Found 0 VPC's to update Redundant State. 2025-06-29 02:57:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85ec942c]) (logid:2629d9be) Found 0 networks to update RvR status. 2025-06-29 02:57:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-ed071ac0]) (logid:b8879a11) DbCollector is running... 2025-06-29 02:57:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cd570c4]) (logid:bcc08e78) No inactive management server node found 2025-06-29 02:57:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cd570c4]) (logid:bcc08e78) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:41,754 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-94647858]) (logid:5b8b4bf2) HA health check task is running... 2025-06-29 02:57:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) ManagementServerCollector is running... 2025-06-29 02:57:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) DB properties were already loaded 2025-06-29 02:57:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:57:42,170 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Execution of process [4062439] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:57:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Exit value of process [4062439] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:57:42,170 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Process [4062439] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:57:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) usage install: null 2025-06-29 02:57:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Sessions active: 0 2025-06-29 02:57:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:57:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062442] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:57:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:57:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062447] for command [/bin/bash -c uname -r ]. 2025-06-29 02:57:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:57:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062448] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:57:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) system memory from /proc: 4887515136 2025-06-29 02:57:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:57:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062453] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:57:42,196 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) free memory from /proc: 507424768 2025-06-29 02:57:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:57:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062458] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:57:42,208 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) used memory from /proc: 761692 2025-06-29 02:57:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:57:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062459] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:57:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:57:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062460] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:57:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:57:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062461] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:57:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:57:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062468] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:57:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:57:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Successfully executed process [4062469] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:57:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:57:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail buffersdirect.count, 120 2025-06-29 02:57:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail buffersdirect.used, 657986 2025-06-29 02:57:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail buffersmapped.count, 7 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail buffersmapped.used, 798602 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail gcPS-Scavenge.count, 491 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail gcPS-Scavenge.time, 3579 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail jvmuptime, 42602322 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memoryheap.committed, 352321536 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memoryheap.init, 77594624 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memoryheap.usage, 0.09578711748843467 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorynon-heap.max, -1 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorynon-heap.usage, -2.20180512E8 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorynon-heap.used, 220180512 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17585518861636898 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.300591320358583 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.Metaspace.usage, 0.9855711596021598 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.PS-Eden-Space.usage, 0.010814888432340802 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.PS-Old-Gen.usage, 0.11969613208302819 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorypools.PS-Survivor-Space.usage, 0.37968902587890624 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorytotal.committed, 577241088 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorytotal.init, 85262336 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorytotal.max, 1908932607 2025-06-29 02:57:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail memorytotal.used, 403086960 2025-06-29 02:57:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail threadsdeadlocks, [] 2025-06-29 02:57:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail threadsnew.count, 0 2025-06-29 02:57:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:57:42,249 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-b8b163df]) (logid:28149b42) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:57:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":170233784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42602247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":402268264,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":182851152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":507424768,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.16],"systemCyclesUsage":[66365747,53806338,4587759240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:57:42,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1419, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:57:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":170233784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42602247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":402268264,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":182851152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":507424768,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.16],"systemCyclesUsage":[66365747,53806338,4587759240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:57:42,249 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:57:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:57:42,254 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-318:[ctx-4f302d55]) (logid:086c70c4) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:57:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":170233784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42602247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":402268264,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":182851152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":507424768,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.16],"systemCyclesUsage":[66365747,53806338,4587759240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:57:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1419, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:57:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":170233784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42602247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":402268264,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":182851152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":507424768,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.16],"systemCyclesUsage":[66365747,53806338,4587759240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:57:42,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2161b29]) (logid:c74c67b1) No inactive management server node found 2025-06-29 02:57:42,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2161b29]) (logid:c74c67b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:44,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16ee97f9]) (logid:df70515e) No inactive management server node found 2025-06-29 02:57:44,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16ee97f9]) (logid:df70515e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:57:45,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-7956: Seq 7-7956: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:57:45,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 7956 2025-06-29 02:57:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:57:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:57:45,755 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b3dfb636]) (logid:eb4d5fd1) HA health check task is running... 2025-06-29 02:57:45,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-094772e9]) (logid:2fc74e8c) No inactive management server node found 2025-06-29 02:57:45,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-094772e9]) (logid:2fc74e8c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-117a01f0]) (logid:201da4c5) VM scheduler.poll is being called at 2025-06-29T00:58:00+0000 2025-06-29 02:57:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-117a01f0]) (logid:201da4c5) Got 0 scheduled jobs to be executed at 2025-06-29T00:58:00+0000 2025-06-29 02:57:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-117a01f0]) (logid:201da4c5) Cleaned up 0 VM scheduled job entries 2025-06-29 02:57:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ac0eb668]) (logid:0eedb01e) VM Operation Thread Running 2025-06-29 02:57:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-499aaf55]) (logid:b87d41cf) ClusterDRS.poll is being called at 2025-06-29T00:58:00+0000 2025-06-29 02:57:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-499aaf55]) (logid:b87d41cf) Removed 0 old drs migration plans 2025-06-29 02:57:47,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:57:47,216 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:57:47,218 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:57:47,220 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:57:47,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1de03afe]) (logid:375f08ec) No inactive management server node found 2025-06-29 02:57:47,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1de03afe]) (logid:375f08ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:48,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8512d4ec]) (logid:7765fbb2) No inactive management server node found 2025-06-29 02:57:48,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8512d4ec]) (logid:7765fbb2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:49,756 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-82dd94a3]) (logid:e068c82a) HA health check task is running... 2025-06-29 02:57:50,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7958: Seq 7-7958: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:57:50,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7958 2025-06-29 02:57:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1277482]) (logid:ab383bf9) No inactive management server node found 2025-06-29 02:57:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1277482]) (logid:ab383bf9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfab8ac0]) (logid:97896d65) No inactive management server node found 2025-06-29 02:57:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfab8ac0]) (logid:97896d65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:53,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b81bf31]) (logid:33c38801) No inactive management server node found 2025-06-29 02:57:53,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b81bf31]) (logid:33c38801) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:53,758 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-50767580]) (logid:287a5baa) HA health check task is running... 2025-06-29 02:57:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-807a2a7f]) (logid:9e05f2d1) No inactive management server node found 2025-06-29 02:57:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-807a2a7f]) (logid:9e05f2d1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:56,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a320ec8]) (logid:ec026417) No inactive management server node found 2025-06-29 02:57:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a320ec8]) (logid:ec026417) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-39a37612]) (logid:8da4e72b) Enabled non-edge zones available for scan: 1 2025-06-29 02:57:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-39a37612]) (logid:8da4e72b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:57:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-39a37612]) (logid:8da4e72b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:57:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c37ea3bd]) (logid:49cb53bc) Enabled non-edge zones available for scan: 1 2025-06-29 02:57:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c37ea3bd]) (logid:49cb53bc) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:57:57,759 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d72aa5f6]) (logid:bc31423b) HA health check task is running... 2025-06-29 02:57:57,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44b55e8a]) (logid:287aa76d) No inactive management server node found 2025-06-29 02:57:57,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44b55e8a]) (logid:287aa76d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:57:57,892 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-53d5e596]) (logid:41817812) AutoScaling Monitor is running... 2025-06-29 02:57:58,008 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-c5d5c823]) (logid:f4e5f972) Starting to remove all volume_stats rows older than [Sat Jun 28 14:57:58 CEST 2025]. 2025-06-29 02:57:58,009 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-c5d5c823]) (logid:f4e5f972) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:57:58 CEST 2025]. 2025-06-29 02:57:59,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fce9ab77]) (logid:508740b2) No inactive management server node found 2025-06-29 02:57:59,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fce9ab77]) (logid:508740b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:00,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7959: Seq 7-7959: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:00,107 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7959 2025-06-29 02:58:00,814 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:58:00,818 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:58:00,819 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) Seq 2-3048092522799696664: Routed from 90520738888109 2025-06-29 02:58:00,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-211cf986]) (logid:9febf175) No inactive management server node found 2025-06-29 02:58:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-211cf986]) (logid:9febf175) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:00,871 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) Seq 1-3048092522799696664: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:58:00,905 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:58:00,905 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) Seq 2-1612007191621929937: Routed from 90520738888109 2025-06-29 02:58:00,949 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d11431e2]) (logid:d63622d7) Seq 2-1612007191621929937: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:58:01,761 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5e5187e3]) (logid:883a6558) HA health check task is running... 2025-06-29 02:58:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5d40b89]) (logid:1a0f599b) No inactive management server node found 2025-06-29 02:58:02,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5d40b89]) (logid:1a0f599b) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a24624a]) (logid:8f200737) No inactive management server node found 2025-06-29 02:58:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a24624a]) (logid:8f200737) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:05,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38a472dc]) (logid:1c2c21cf) No inactive management server node found 2025-06-29 02:58:05,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38a472dc]) (logid:1c2c21cf) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:05,566 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) StorageCollector is running... 2025-06-29 02:58:05,573 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:58:05,573 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:58:05,573 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:58:05,574 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-6717118844223095474: Routed from 90520738888109 2025-06-29 02:58:05,629 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 4-6717118844223095474: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:58:05,633 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:58:05,633 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:58:05,633 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:58:05,634 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-3048092522799696665: Routed from 90520738888109 2025-06-29 02:58:05,682 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 1-3048092522799696665: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:58:05,704 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:58:05,704 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:58:05,704 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:58:05,705 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-1612007191621929938: Routed from 90520738888109 2025-06-29 02:58:05,716 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-1612007191621929938: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:58:05,725 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:58:05,725 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:58:05,726 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:58:05,726 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-3048092522799696666: Routed from 90520738888109 2025-06-29 02:58:05,762 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4772ad52]) (logid:cacb4133) HA health check task is running... 2025-06-29 02:58:06,778 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 1-3048092522799696666: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:58:06,780 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:58:06,780 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:58:06,780 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:58:06,780 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-1612007191621929939: Routed from 90520738888109 2025-06-29 02:58:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e384869]) (logid:49c05232) No inactive management server node found 2025-06-29 02:58:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e384869]) (logid:49c05232) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:07,843 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb50d88e]) (logid:57e71ae8) Seq 2-1612007191621929939: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:58:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2597531b]) (logid:ac815f5f) No inactive management server node found 2025-06-29 02:58:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2597531b]) (logid:ac815f5f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:09,763 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-072057d9]) (logid:bc94bd91) HA health check task is running... 2025-06-29 02:58:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b94325f9]) (logid:5e5aec35) No inactive management server node found 2025-06-29 02:58:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b94325f9]) (logid:5e5aec35) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:10,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7960: Seq 7-7960: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:10,117 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7960 2025-06-29 02:58:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-21ba0038]) (logid:1194f1a5) Found 0 routers to update status. 2025-06-29 02:58:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-21ba0038]) (logid:1194f1a5) Found 0 VPC's to update Redundant State. 2025-06-29 02:58:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-21ba0038]) (logid:1194f1a5) Found 0 networks to update RvR status. 2025-06-29 02:58:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-065ecbef]) (logid:3f703074) No inactive management server node found 2025-06-29 02:58:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-065ecbef]) (logid:3f703074) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:12,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72fcaeb9]) (logid:f22320a1) No inactive management server node found 2025-06-29 02:58:12,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72fcaeb9]) (logid:f22320a1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:13,220 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-8a4b3be9]) (logid:f3d9d3dd) Starting to remove all vm_stats rows older than [Sat Jun 28 14:58:13 CEST 2025]. 2025-06-29 02:58:13,243 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-8a4b3be9]) (logid:f3d9d3dd) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:58:13 CEST 2025]. 2025-06-29 02:58:13,765 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a6676af6]) (logid:edba86ce) HA health check task is running... 2025-06-29 02:58:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bce4be9]) (logid:8840542c) No inactive management server node found 2025-06-29 02:58:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bce4be9]) (logid:8840542c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:15,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb69e57]) (logid:9c77bd0a) No inactive management server node found 2025-06-29 02:58:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb69e57]) (logid:9c77bd0a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-059610d6]) (logid:78518684) Resetting hosts suitable for reconnect 2025-06-29 02:58:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-059610d6]) (logid:78518684) Completed resetting hosts suitable for reconnect 2025-06-29 02:58:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-059610d6]) (logid:78518684) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:58:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-059610d6]) (logid:78518684) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:58:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-059610d6]) (logid:78518684) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:58:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-059610d6]) (logid:78518684) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:58:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f72e8f6]) (logid:bd673039) No inactive management server node found 2025-06-29 02:58:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f72e8f6]) (logid:bd673039) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:17,766 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-00f74f27]) (logid:6ca0a90e) HA health check task is running... 2025-06-29 02:58:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-634c51a6]) (logid:9dbba1c7) No inactive management server node found 2025-06-29 02:58:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-634c51a6]) (logid:9dbba1c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:20,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7961: Seq 7-7961: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:20,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7961 2025-06-29 02:58:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2870eff]) (logid:5aa7aba5) No inactive management server node found 2025-06-29 02:58:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2870eff]) (logid:5aa7aba5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:21,767 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b240f265]) (logid:81062b9c) HA health check task is running... 2025-06-29 02:58:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e0cf5a7]) (logid:2a21db40) No inactive management server node found 2025-06-29 02:58:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e0cf5a7]) (logid:2a21db40) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:23,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5225ac31]) (logid:0ff18a46) No inactive management server node found 2025-06-29 02:58:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5225ac31]) (logid:0ff18a46) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:24,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04e76b29]) (logid:92304f0d) No inactive management server node found 2025-06-29 02:58:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04e76b29]) (logid:92304f0d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:25,769 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7657b855]) (logid:9b77071f) HA health check task is running... 2025-06-29 02:58:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97575dc9]) (logid:deca2dc8) No inactive management server node found 2025-06-29 02:58:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97575dc9]) (logid:deca2dc8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3c3f12ef]) (logid:081275bb) Enabled non-edge zones available for scan: 1 2025-06-29 02:58:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3c3f12ef]) (logid:081275bb) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:58:26,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3c3f12ef]) (logid:081275bb) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:58:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3fa1fe7e]) (logid:834ff3fe) Enabled non-edge zones available for scan: 1 2025-06-29 02:58:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3fa1fe7e]) (logid:834ff3fe) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:58:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a96aa41b]) (logid:81e6ece2) No inactive management server node found 2025-06-29 02:58:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a96aa41b]) (logid:81e6ece2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:29,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3707b9cd]) (logid:9bd3a2ae) No inactive management server node found 2025-06-29 02:58:29,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3707b9cd]) (logid:9bd3a2ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:29,770 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1f59ac62]) (logid:b82c0a9e) HA health check task is running... 2025-06-29 02:58:29,980 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:58:29,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:58:29,982 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) Seq 2-3048092522799696667: Routed from 90520738888109 2025-06-29 02:58:29,984 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) Seq 1-3048092522799696667: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:58:29,985 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:58:29,986 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) Seq 2-1612007191621929940: Routed from 90520738888109 2025-06-29 02:58:29,989 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-de02d224]) (logid:ce084550) Seq 2-1612007191621929940: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:58:30,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7962: Seq 7-7962: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:30,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7962 2025-06-29 02:58:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2e4262f]) (logid:a8a6cb47) No inactive management server node found 2025-06-29 02:58:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2e4262f]) (logid:a8a6cb47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce9eadcc]) (logid:74854242) No inactive management server node found 2025-06-29 02:58:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce9eadcc]) (logid:74854242) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:33,771 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d8504055]) (logid:4c3636d8) HA health check task is running... 2025-06-29 02:58:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f0fda5d]) (logid:a0cc6534) No inactive management server node found 2025-06-29 02:58:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f0fda5d]) (logid:a0cc6534) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:58:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:58:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:58:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:58:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-169df031]) (logid:0eaf25f7) No inactive management server node found 2025-06-29 02:58:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-169df031]) (logid:0eaf25f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e7bf5dd]) (logid:723c93f9) No inactive management server node found 2025-06-29 02:58:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e7bf5dd]) (logid:723c93f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:37,772 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1e389356]) (logid:4219ff48) HA health check task is running... 2025-06-29 02:58:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fee1afcf]) (logid:55998135) No inactive management server node found 2025-06-29 02:58:38,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fee1afcf]) (logid:55998135) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a166c49]) (logid:07104d5f) No inactive management server node found 2025-06-29 02:58:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a166c49]) (logid:07104d5f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:40,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7963: Seq 7-7963: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:40,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7963 2025-06-29 02:58:40,815 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-4bc771a4]) (logid:7ade4628) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 02:58:40,816 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-4bc771a4]) (logid:7ade4628) Snapshot scheduler is being called at [2025-06-29T00:58:40+0000]. 2025-06-29 02:58:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-4bc771a4]) (logid:7ade4628) There are [0] scheduled snapshots to be executed at [2025-06-29T00:58:40+0000]. 2025-06-29 02:58:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-afe17fb5]) (logid:b5f8904a) External devices are not used. Skipping external device usage collection 2025-06-29 02:58:40,946 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-347f91f8]) (logid:0b8f4084) checking health of usage server 2025-06-29 02:58:40,947 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-347f91f8]) (logid:0b8f4084) usage server running? false, heartbeat: null 2025-06-29 02:58:40,951 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-347f91f8]) (logid:0b8f4084) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 02:58:40,953 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-347f91f8]) (logid:0b8f4084) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 02:58:40,979 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-a061deaa]) (logid:0853bca1) Backup backup.poll is being called at 2025-06-29T00:58:40+0000 2025-06-29 02:58:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c03343e2]) (logid:453ebb9e) Found 0 routers to update status. 2025-06-29 02:58:41,000 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-bfcf2ec4]) (logid:a5e60b8a) Found 0 running routers. 2025-06-29 02:58:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c03343e2]) (logid:453ebb9e) Found 0 VPC's to update Redundant State. 2025-06-29 02:58:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c03343e2]) (logid:453ebb9e) Found 0 networks to update RvR status. 2025-06-29 02:58:41,018 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Found 1 running routers. 2025-06-29 02:58:41,024 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 02:58:41,026 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Seq 2-1612007191621929941: Routed from 90520738888109 2025-06-29 02:58:41,026 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Seq 2-1612007191621929941: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:41,043 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-80ddff08]) (logid:db7e32d1) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 02:58:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-d2f5731e]) (logid:3bd3b053) DbCollector is running... 2025-06-29 02:58:41,279 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-4f7b1bd9]) (logid:125374ec) Started resource counters recalculation periodic task. 2025-06-29 02:58:41,283 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-4f7b1bd9]) (logid:125374ec) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 02:58:41,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efcf8467]) (logid:ddb2d7d9) No inactive management server node found 2025-06-29 02:58:41,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efcf8467]) (logid:ddb2d7d9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:58:41,579 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-1612007191621929941: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:41,579 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Seq 2-1612007191621929941: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 02:58:41,579 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 02:58:41,583 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 02:58:41,628 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 02:58:41,630 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Seq 2-1612007191621929942: Routed from 90520738888109 2025-06-29 02:58:41,631 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Seq 2-1612007191621929942: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:41,667 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-4f7b1bd9]) (logid:125374ec) Finished resource counters recalculation periodic task. 2025-06-29 02:58:41,774 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-91d0f48b]) (logid:cc75fdd2) HA health check task is running... 2025-06-29 02:58:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) ManagementServerCollector is running... 2025-06-29 02:58:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) DB properties were already loaded 2025-06-29 02:58:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:58:42,167 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Execution of process [4062487] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:58:42,167 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Exit value of process [4062487] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:58:42,167 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Process [4062487] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:58:42,167 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) usage install: null 2025-06-29 02:58:42,167 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Sessions active: 0 2025-06-29 02:58:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:58:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062490] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:58:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:58:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062495] for command [/bin/bash -c uname -r ]. 2025-06-29 02:58:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:58:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062496] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:58:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) system memory from /proc: 4887515136 2025-06-29 02:58:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:58:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062501] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:58:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) free memory from /proc: 503078912 2025-06-29 02:58:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:58:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062506] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:58:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) used memory from /proc: 761692 2025-06-29 02:58:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:58:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062507] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:58:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:58:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062508] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:58:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:58:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062509] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:58:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:58:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062516] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:58:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:58:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Successfully executed process [4062517] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail buffersdirect.capacity, 657986 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail buffersdirect.count, 120 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail buffersdirect.used, 657986 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail buffersmapped.count, 7 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail buffersmapped.used, 798602 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail gcPS-Scavenge.count, 492 2025-06-29 02:58:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail gcPS-Scavenge.time, 3584 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail jvmuptime, 42662321 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memoryheap.committed, 352321536 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memoryheap.init, 77594624 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memoryheap.usage, 0.10285971499314449 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorynon-heap.max, -1 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorynon-heap.usage, -2.2018384E8 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorynon-heap.used, 220183840 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1758656025059984 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.300607983137268 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.Metaspace.usage, 0.9855711596021598 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.PS-Eden-Space.usage, 0.027906610595334887 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.PS-Old-Gen.usage, 0.11971901747299524 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorypools.PS-Survivor-Space.usage, 0.5312530517578125 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorytotal.committed, 577241088 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorytotal.init, 85262336 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorytotal.max, 1908932607 2025-06-29 02:58:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail memorytotal.used, 416590848 2025-06-29 02:58:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail threadsdeadlocks, [] 2025-06-29 02:58:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail threadsnew.count, 0 2025-06-29 02:58:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) not storing detail threadstimed_waiting.count, 85 2025-06-29 02:58:42,247 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-97f82e35]) (logid:35de0a1d) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:58:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":156727512,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42662245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":415777864,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196352264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":503078912,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.07,0.15],"systemCyclesUsage":[66366113,53806658,4587782291],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:58:42,248 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1421, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:58:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":156727512,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42662245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":415777864,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196352264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":503078912,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.07,0.15],"systemCyclesUsage":[66366113,53806658,4587782291],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:58:42,248 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:58:42,253 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-318:[ctx-ec51511e]) (logid:5f9b350e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:58:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":156727512,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42662245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":415777864,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196352264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":503078912,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.07,0.15],"systemCyclesUsage":[66366113,53806658,4587782291],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:58:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 02:58:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1421, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:58:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":156727512,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42662245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":415777864,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196352264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":503078912,"systemMemoryUsed":761692,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.07,0.15],"systemCyclesUsage":[66366113,53806658,4587782291],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:58:42,836 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48de9cd6]) (logid:f5d648fd) No inactive management server node found 2025-06-29 02:58:42,836 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48de9cd6]) (logid:f5d648fd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:58:43,174 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 2-1612007191621929942: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.7420e693-1e72-475c-b72f-4f95b3008678","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:43,174 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b1894352]) (logid:85ae4cf7) Seq 2-1612007191621929942: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 02:58:43,177 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 02:58:43,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 02:58:43,181 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Seq 2-1612007191621929943: Routed from 90520738888109 2025-06-29 02:58:43,182 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Seq 2-1612007191621929943: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:43,618 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 2-1612007191621929943: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "9.30643081665039", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "18.183231353759766", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "27.06432342529297", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "34.93332862854004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158621371", "lastRunDuration": "49.51214790344238", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158621445", "lastRunDuration": "8071.237325668335", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158629561", "lastRunDuration": "55.75132369995117", "message": "Memory Usage within limits with current at 50.8663%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158629617", "lastRunDuration": "3227.2770404815674", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158632893", "lastRunDuration": "39.96610641479492", "message": "Sufficient free space is 2571.5625 MB"}, "lastRun": {"start": "2025-06-29 00:57:01.227393", "end": "2025-06-29 00:57:12.933338", "duration": "11.705945014953613"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158201993", "lastRunDuration": "47.601938247680664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158202041", "lastRunDuration": "52.01077461242676", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158202208", "lastRunDuration": "51.415205001831055", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158202368", "lastRunDuration": "48.478126525878906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:50:01.991127", "end": "2025-06-29 00:50:02.462928", "duration": "0.4718008041381836"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "9.30643081665039", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "18.183231353759766", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "27.06432342529297", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "34.93332862854004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158621371", "lastRunDuration": "49.51214790344238", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158621445", "lastRunDuration": "8071.237325668335", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158629561", "lastRunDuration": "55.75132369995117", "message": "Memory Usage within limits with current at 50.8663%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158629617", "lastRunDuration": "3227.2770404815674", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158632893", "lastRunDuration": "39.96610641479492", "message": "Sufficient free space is 2571.5625 MB"}, "lastRun": {"start": "2025-06-29 00:57:01.227393", "end": "2025-06-29 00:57:12.933338", "duration": "11.705945014953613"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158201993", "lastRunDuration": "47.601938247680664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158202041", "lastRunDuration": "52.01077461242676", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158202208", "lastRunDuration": "51.415205001831055", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158202368", "lastRunDuration": "48.478126525878906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:50:01.991127", "end": "2025-06-29 00:50:02.462928", "duration": "0.4718008041381836"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:43,618 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Seq 2-1612007191621929943: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 02:58:43,618 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "9.30643081665039", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "18.183231353759766", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "27.06432342529297", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "34.93332862854004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158621371", "lastRunDuration": "49.51214790344238", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158621445", "lastRunDuration": "8071.237325668335", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158629561", "lastRunDuration": "55.75132369995117", "message": "Memory Usage within limits with current at 50.8663%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158629617", "lastRunDuration": "3227.2770404815674", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158632893", "lastRunDuration": "39.96610641479492", "message": "Sufficient free space is 2571.5625 MB"}, "lastRun": {"start": "2025-06-29 00:57:01.227393", "end": "2025-06-29 00:57:12.933338", "duration": "11.705945014953613"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158201993", "lastRunDuration": "47.601938247680664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158202041", "lastRunDuration": "52.01077461242676", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158202208", "lastRunDuration": "51.415205001831055", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158202368", "lastRunDuration": "48.478126525878906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:50:01.991127", "end": "2025-06-29 00:50:02.462928", "duration": "0.4718008041381836"}}} 2025-06-29 02:58:43,667 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "9.30643081665039", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "18.183231353759766", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "27.06432342529297", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751158621229", "lastRunDuration": "34.93332862854004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751158621371", "lastRunDuration": "49.51214790344238", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751158621445", "lastRunDuration": "8071.237325668335", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751158629561", "lastRunDuration": "55.75132369995117", "message": "Memory Usage within limits with current at 50.8663%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751158629617", "lastRunDuration": "3227.2770404815674", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751158632893", "lastRunDuration": "39.96610641479492", "message": "Sufficient free space is 2571.5625 MB"}, "lastRun": {"start": "2025-06-29 00:57:01.227393", "end": "2025-06-29 00:57:12.933338", "duration": "11.705945014953613"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158201993", "lastRunDuration": "47.601938247680664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158202041", "lastRunDuration": "52.01077461242676", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158202208", "lastRunDuration": "51.415205001831055", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158202368", "lastRunDuration": "48.478126525878906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 00:50:01.991127", "end": "2025-06-29 00:50:02.462928", "duration": "0.4718008041381836"}}} 2025-06-29 02:58:43,669 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found check types executed on VR basic, start: 2025-06-29 00:57:01.227393, end: 2025-06-29 00:57:12.933338, duration: 11.705945014953613 2025-06-29 02:58:43,677 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4aadb879- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 02:57:01 CEST 2025, details: service is running which took running duration (ms) 9.30643081665039 2025-06-29 02:58:43,688 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@13622d7e- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 02:57:01 CEST 2025, details: service is running which took running duration (ms) 18.183231353759766 2025-06-29 02:58:43,699 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@29b4b7d7- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 02:57:01 CEST 2025, details: service is running which took running duration (ms) 27.06432342529297 2025-06-29 02:58:43,709 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@395f59bd- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 02:57:01 CEST 2025, details: service is running which took running duration (ms) 34.93332862854004 2025-06-29 02:58:43,720 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1e96409f- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 02:57:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 49.51214790344238 2025-06-29 02:58:43,729 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1c1d87ee- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 02:57:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8071.237325668335 2025-06-29 02:58:43,739 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@101b81b5- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 02:57:09 CEST 2025, details: Memory Usage within limits with current at 50.8663% which took running duration (ms) 55.75132369995117 2025-06-29 02:58:43,749 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@534aa0ba- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 02:57:09 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3227.2770404815674 2025-06-29 02:58:43,761 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7ff490f4- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 02:57:12 CEST 2025, details: Sufficient free space is 2571.5625 MB which took running duration (ms) 39.96610641479492 2025-06-29 02:58:43,761 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found check types executed on VR advanced, start: 2025-06-29 00:50:01.991127, end: 2025-06-29 00:50:02.462928, duration: 0.4718008041381836 2025-06-29 02:58:43,771 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@f265a13- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 02:50:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 47.601938247680664 2025-06-29 02:58:43,781 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@43fc646b- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 02:50:02 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 52.01077461242676 2025-06-29 02:58:43,792 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@453543de- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 02:50:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 51.415205001831055 2025-06-29 02:58:43,803 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-d0983ac2]) (logid:225678ef) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1e1d08bc- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 02:50:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 48.478126525878906 2025-06-29 02:58:44,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1095ffc3]) (logid:28bdc07b) No inactive management server node found 2025-06-29 02:58:44,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1095ffc3]) (logid:28bdc07b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:58:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:58:45,775 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7b685830]) (logid:45d5f7e3) HA health check task is running... 2025-06-29 02:58:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e33957b]) (logid:4e831827) No inactive management server node found 2025-06-29 02:58:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e33957b]) (logid:4e831827) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9ac69fdd]) (logid:0f94f23f) VM scheduler.poll is being called at 2025-06-29T00:59:00+0000 2025-06-29 02:58:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9ac69fdd]) (logid:0f94f23f) Got 0 scheduled jobs to be executed at 2025-06-29T00:59:00+0000 2025-06-29 02:58:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9ac69fdd]) (logid:0f94f23f) Cleaned up 0 VM scheduled job entries 2025-06-29 02:58:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-326ab336]) (logid:ab649672) VM Operation Thread Running 2025-06-29 02:58:46,181 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-3:[ctx-6a2565e6]) (logid:adb8ee36) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 02:58:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-983f1ace]) (logid:4abff481) ClusterDRS.poll is being called at 2025-06-29T00:59:00+0000 2025-06-29 02:58:46,336 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-983f1ace]) (logid:4abff481) Removed 0 old drs migration plans 2025-06-29 02:58:47,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:58:47,218 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:58:47,219 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:58:47,222 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:58:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b473aa6]) (logid:04118355) No inactive management server node found 2025-06-29 02:58:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b473aa6]) (logid:04118355) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67fa3194]) (logid:cf9f4c17) No inactive management server node found 2025-06-29 02:58:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67fa3194]) (logid:cf9f4c17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:49,776 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-153266fa]) (logid:2122692b) HA health check task is running... 2025-06-29 02:58:50,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7965: Seq 7-7965: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:50,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7965 2025-06-29 02:58:50,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59c6a2e0]) (logid:c0128c21) No inactive management server node found 2025-06-29 02:58:50,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59c6a2e0]) (logid:c0128c21) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:51,032 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-a717eb95]) (logid:4eb5544b) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 02:58:51,041 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-a717eb95]) (logid:4eb5544b) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 02:58:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bdec35c]) (logid:f9596458) No inactive management server node found 2025-06-29 02:58:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bdec35c]) (logid:f9596458) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e424d60f]) (logid:28bb68e6) No inactive management server node found 2025-06-29 02:58:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e424d60f]) (logid:28bb68e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:53,778 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-90611773]) (logid:b67847ee) HA health check task is running... 2025-06-29 02:58:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebdcf981]) (logid:8682a608) No inactive management server node found 2025-06-29 02:58:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebdcf981]) (logid:8682a608) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:56,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 02:58:56,224 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Seq 2-3048092522799696668: Routed from 90520738888109 2025-06-29 02:58:56,339 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1664973]) (logid:71f5006a) No inactive management server node found 2025-06-29 02:58:56,339 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1664973]) (logid:71f5006a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f7790aaf]) (logid:7fa79a02) Enabled non-edge zones available for scan: 1 2025-06-29 02:58:56,766 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f7790aaf]) (logid:7fa79a02) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:58:56,767 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f7790aaf]) (logid:7fa79a02) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:58:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6a06e1dd]) (logid:761c0de3) Enabled non-edge zones available for scan: 1 2025-06-29 02:58:56,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6a06e1dd]) (logid:761c0de3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:58:57,779 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0e2d31ed]) (logid:6d3282fb) HA health check task is running... 2025-06-29 02:58:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7933c81]) (logid:68d9cf1f) No inactive management server node found 2025-06-29 02:58:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7933c81]) (logid:68d9cf1f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:58:57,894 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-db8583c7]) (logid:b2a6d7ff) AutoScaling Monitor is running... 2025-06-29 02:58:58,011 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-d7397158]) (logid:3388111b) Starting to remove all volume_stats rows older than [Sat Jun 28 14:58:58 CEST 2025]. 2025-06-29 02:58:58,012 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-d7397158]) (logid:3388111b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:58:58 CEST 2025]. 2025-06-29 02:58:59,291 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Seq 1-3048092522799696668: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 02:58:59,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 02:58:59,299 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 2-1612007191621926913: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:58:59,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 02:58:59,323 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Seq 2-1612007191621929944: Routed from 90520738888109 2025-06-29 02:58:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4adfe48d]) (logid:cf23dabd) No inactive management server node found 2025-06-29 02:58:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4adfe48d]) (logid:cf23dabd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:59:00,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7966: Seq 7-7966: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:00,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7966 2025-06-29 02:59:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b53fa56]) (logid:f3b2646f) No inactive management server node found 2025-06-29 02:59:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b53fa56]) (logid:f3b2646f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:01,005 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 02:59:01,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:59:01,011 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) Seq 2-3048092522799696669: Routed from 90520738888109 2025-06-29 02:59:01,052 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) Seq 1-3048092522799696669: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:59:01,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 02:59:01,087 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) Seq 2-1612007191621929945: Routed from 90520738888109 2025-06-29 02:59:01,142 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-29d103c7]) (logid:50ca16ef) Seq 2-1612007191621929945: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 02:59:01,781 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a092f7ef]) (logid:bbb79ecd) HA health check task is running... 2025-06-29 02:59:02,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73b80f29]) (logid:b59af339) No inactive management server node found 2025-06-29 02:59:02,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73b80f29]) (logid:b59af339) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d4f43f3]) (logid:dfc81811) No inactive management server node found 2025-06-29 02:59:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d4f43f3]) (logid:dfc81811) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:04,417 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Seq 2-1612007191621929944: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 02:59:04,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43976230]) (logid:d4036924) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 02:59:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21f66351]) (logid:b37da923) No inactive management server node found 2025-06-29 02:59:05,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21f66351]) (logid:b37da923) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:59:05,782 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-587f3325]) (logid:14023a4f) HA health check task is running... 2025-06-29 02:59:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e906dc35]) (logid:44488944) No inactive management server node found 2025-06-29 02:59:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e906dc35]) (logid:44488944) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:59:07,843 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) StorageCollector is running... 2025-06-29 02:59:07,849 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:59:07,849 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:59:07,849 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:59:07,850 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-6717118844223095475: Routed from 90520738888109 2025-06-29 02:59:07,904 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 4-6717118844223095475: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:59:07,908 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:59:07,908 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:59:07,908 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:59:07,909 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-1612007191621929946: Routed from 90520738888109 2025-06-29 02:59:07,920 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-1612007191621929946: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:59:07,942 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:59:07,942 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:59:07,942 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:59:07,943 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-1612007191621929947: Routed from 90520738888109 2025-06-29 02:59:08,036 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-1612007191621929947: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:59:08,046 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:59:08,046 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:59:08,046 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:59:08,047 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-3048092522799696670: Routed from 90520738888109 2025-06-29 02:59:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3be9e8fd]) (logid:8447de80) No inactive management server node found 2025-06-29 02:59:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3be9e8fd]) (logid:8447de80) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:09,099 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 1-3048092522799696670: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:59:09,101 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 02:59:09,101 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 02:59:09,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 02:59:09,102 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-1612007191621929948: Routed from 90520738888109 2025-06-29 02:59:09,783 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-02dc4f37]) (logid:f1e6272d) HA health check task is running... 2025-06-29 02:59:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7357ac81]) (logid:ce760a04) No inactive management server node found 2025-06-29 02:59:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7357ac81]) (logid:ce760a04) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:10,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7967: Seq 7-7967: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:10,119 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7967 2025-06-29 02:59:10,165 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64799e62]) (logid:d0031281) Seq 2-1612007191621929948: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 02:59:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-03eb4375]) (logid:14d923cc) Found 0 routers to update status. 2025-06-29 02:59:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-03eb4375]) (logid:14d923cc) Found 0 VPC's to update Redundant State. 2025-06-29 02:59:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-03eb4375]) (logid:14d923cc) Found 0 networks to update RvR status. 2025-06-29 02:59:11,066 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Running Capacity Checker ... 2025-06-29 02:59:11,066 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) recalculating system capacity 2025-06-29 02:59:11,066 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Executing cpu/ram capacity update 2025-06-29 02:59:11,071 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:11,085 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 02:59:11,085 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 02:59:11,086 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 02:59:11,106 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:11,107 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:11,111 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:11,112 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 02:59:11,113 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 02:59:11,113 DEBUG [c.c.c.CapacityManagerImpl] (pool-312-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 02:59:11,121 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Done executing cpu/ram capacity update 2025-06-29 02:59:11,121 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Executing storage capacity update 2025-06-29 02:59:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:59:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 02:59:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 02:59:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 02:59:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 02:59:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 02:59:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 02:59:11,142 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 02:59:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:59:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 02:59:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 02:59:11,161 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 02:59:11,165 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 02:59:11,166 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 02:59:11,166 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 02:59:11,170 DEBUG [c.c.s.StorageManagerImpl] (pool-313-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 02:59:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Done executing storage capacity update 2025-06-29 02:59:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Executing capacity updates for public ip and Vlans 2025-06-29 02:59:11,187 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Done capacity updates for public ip and Vlans 2025-06-29 02:59:11,187 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Executing capacity updates for private ip 2025-06-29 02:59:11,192 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Done executing capacity updates for private ip 2025-06-29 02:59:11,192 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Done recalculating system capacity 2025-06-29 02:59:11,227 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-dbeeeb25]) (logid:6f42dc7b) Done running Capacity Checker ... 2025-06-29 02:59:11,258 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b11e3c78]) (logid:601f308e) Checking if any host reservation can be released ... 2025-06-29 02:59:11,265 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b11e3c78]) (logid:601f308e) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:11,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b11e3c78]) (logid:601f308e) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:11,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b11e3c78]) (logid:601f308e) Done running HostReservationReleaseChecker ... 2025-06-29 02:59:11,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e7f14a3]) (logid:583e447d) No inactive management server node found 2025-06-29 02:59:11,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e7f14a3]) (logid:583e447d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-025649c0]) (logid:60776ddc) No inactive management server node found 2025-06-29 02:59:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-025649c0]) (logid:60776ddc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:13,244 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-9081586d]) (logid:fc6b565c) Starting to remove all vm_stats rows older than [Sat Jun 28 14:59:13 CEST 2025]. 2025-06-29 02:59:13,270 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-9081586d]) (logid:fc6b565c) Removed a total of [5] vm_stats rows older than [Sat Jun 28 14:59:13 CEST 2025]. 2025-06-29 02:59:13,784 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-bb46919b]) (logid:3e2aba60) HA health check task is running... 2025-06-29 02:59:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ce6eaee]) (logid:0d38057f) No inactive management server node found 2025-06-29 02:59:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ce6eaee]) (logid:0d38057f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46ba2cbd]) (logid:d2ecc096) No inactive management server node found 2025-06-29 02:59:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46ba2cbd]) (logid:d2ecc096) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:17,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f77089f1]) (logid:00eb6cc4) No inactive management server node found 2025-06-29 02:59:17,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f77089f1]) (logid:00eb6cc4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:17,786 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c539d01d]) (logid:32947072) HA health check task is running... 2025-06-29 02:59:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac978228]) (logid:6640b073) No inactive management server node found 2025-06-29 02:59:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac978228]) (logid:6640b073) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:20,099 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7968: Seq 7-7968: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:20,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7968 2025-06-29 02:59:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84934466]) (logid:5b90e3fc) No inactive management server node found 2025-06-29 02:59:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84934466]) (logid:5b90e3fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:21,787 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d6f4f663]) (logid:c4abcb73) HA health check task is running... 2025-06-29 02:59:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc3d189c]) (logid:99d69a47) No inactive management server node found 2025-06-29 02:59:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc3d189c]) (logid:99d69a47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:23,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59890789]) (logid:085ff659) No inactive management server node found 2025-06-29 02:59:23,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59890789]) (logid:085ff659) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:24,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9600550]) (logid:ba617c98) No inactive management server node found 2025-06-29 02:59:24,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9600550]) (logid:ba617c98) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:25,788 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-763892f3]) (logid:d8ce7783) HA health check task is running... 2025-06-29 02:59:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7f5b54e]) (logid:5d516bdd) No inactive management server node found 2025-06-29 02:59:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7f5b54e]) (logid:5d516bdd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:59:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-651dd135]) (logid:cd942fff) Enabled non-edge zones available for scan: 1 2025-06-29 02:59:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-651dd135]) (logid:cd942fff) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:59:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-651dd135]) (logid:cd942fff) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:59:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2c4a7e8a]) (logid:d74fbeb6) Enabled non-edge zones available for scan: 1 2025-06-29 02:59:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2c4a7e8a]) (logid:d74fbeb6) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:59:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b857027c]) (logid:9a9bfe9d) No inactive management server node found 2025-06-29 02:59:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b857027c]) (logid:9a9bfe9d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ef4eb22]) (logid:a32ad3ec) No inactive management server node found 2025-06-29 02:59:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ef4eb22]) (logid:a32ad3ec) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:59:29,790 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7f783294]) (logid:f944440d) HA health check task is running... 2025-06-29 02:59:29,992 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) HostStatsCollector is running to process 2 UP hosts 2025-06-29 02:59:29,994 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:59:29,995 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) Seq 2-3048092522799696671: Routed from 90520738888109 2025-06-29 02:59:29,997 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) Seq 1-3048092522799696671: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:59:29,998 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 02:59:29,999 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) Seq 2-1612007191621929949: Routed from 90520738888109 2025-06-29 02:59:30,003 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-9464ac69]) (logid:54965c56) Seq 2-1612007191621929949: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 02:59:30,099 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7969: Seq 7-7969: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:30,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7969 2025-06-29 02:59:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5480a64f]) (logid:b7bc58c3) No inactive management server node found 2025-06-29 02:59:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5480a64f]) (logid:b7bc58c3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:32,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f8a19ab]) (logid:adac233f) No inactive management server node found 2025-06-29 02:59:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f8a19ab]) (logid:adac233f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:33,791 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-69ecd84b]) (logid:88822973) HA health check task is running... 2025-06-29 02:59:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dba53f9]) (logid:90924e65) No inactive management server node found 2025-06-29 02:59:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dba53f9]) (logid:90924e65) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 02:59:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 02:59:33,994 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 02:59:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-606ab80b]) (logid:f069930e) No inactive management server node found 2025-06-29 02:59:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-606ab80b]) (logid:f069930e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5c15b63]) (logid:3d271372) No inactive management server node found 2025-06-29 02:59:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5c15b63]) (logid:3d271372) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:37,792 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-215a7131]) (logid:9a00bbd1) HA health check task is running... 2025-06-29 02:59:38,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69a1e55d]) (logid:0fe4cd4f) No inactive management server node found 2025-06-29 02:59:38,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69a1e55d]) (logid:0fe4cd4f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75744cd0]) (logid:bd68d21d) No inactive management server node found 2025-06-29 02:59:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75744cd0]) (logid:bd68d21d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:40,099 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7970: Seq 7-7970: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:40,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7970 2025-06-29 02:59:40,127 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 1-3048092522799693825: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3e418422]) (logid:4b8fd69e) Found 0 routers to update status. 2025-06-29 02:59:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3e418422]) (logid:4b8fd69e) Found 0 VPC's to update Redundant State. 2025-06-29 02:59:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3e418422]) (logid:4b8fd69e) Found 0 networks to update RvR status. 2025-06-29 02:59:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-1f111093]) (logid:3d48a4ef) DbCollector is running... 2025-06-29 02:59:41,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df47f350]) (logid:b25eff8e) No inactive management server node found 2025-06-29 02:59:41,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df47f350]) (logid:b25eff8e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:41,793 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-85216c14]) (logid:3769fe65) HA health check task is running... 2025-06-29 02:59:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) ManagementServerCollector is running... 2025-06-29 02:59:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) DB properties were already loaded 2025-06-29 02:59:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 02:59:42,169 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Execution of process [4062531] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 02:59:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Exit value of process [4062531] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 02:59:42,170 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Process [4062531] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 02:59:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) usage install: null 2025-06-29 02:59:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Sessions active: 0 2025-06-29 02:59:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:59:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062534] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 02:59:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c uname -r ]. 2025-06-29 02:59:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062539] for command [/bin/bash -c uname -r ]. 2025-06-29 02:59:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:59:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062540] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:59:42,190 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) system memory from /proc: 4887515136 2025-06-29 02:59:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:59:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062545] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 02:59:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) free memory from /proc: 504147968 2025-06-29 02:59:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:59:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062550] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 02:59:42,212 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) used memory from /proc: 761820 2025-06-29 02:59:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 02:59:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062551] for command [/bin/bash -c uptime -s ]. 2025-06-29 02:59:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:59:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062552] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 02:59:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:59:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062553] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 02:59:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:59:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062560] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 02:59:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:59:42,253 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Successfully executed process [4062561] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 02:59:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail buffersdirect.capacity, 658029 2025-06-29 02:59:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail buffersdirect.count, 121 2025-06-29 02:59:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail buffersdirect.used, 658029 2025-06-29 02:59:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail buffersmapped.capacity, 798602 2025-06-29 02:59:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail buffersmapped.count, 7 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail buffersmapped.used, 798602 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail gcPS-Scavenge.count, 494 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail gcPS-Scavenge.time, 3598 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail jvmuptime, 42722334 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memoryheap.committed, 352321536 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memoryheap.init, 77594624 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memoryheap.usage, 0.09643886181654035 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorynon-heap.committed, 224919552 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorynon-heap.init, 7667712 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorynon-heap.max, -1 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorynon-heap.usage, -2.20284864E8 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorynon-heap.used, 220284864 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1758656025059984 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3012547072349785 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.Metaspace.usage, 0.9857163709504806 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.PS-Eden-Space.usage, 0.004727241947965802 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.PS-Old-Gen.usage, 0.11975906690543757 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorypools.PS-Survivor-Space.usage, 0.9424414383737665 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorytotal.committed, 577241088 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorytotal.init, 85262336 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorytotal.max, 1908932607 2025-06-29 02:59:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail memorytotal.used, 404431584 2025-06-29 02:59:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail threadsdeadlocks, [] 2025-06-29 02:59:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail threadsnew.count, 0 2025-06-29 02:59:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) not storing detail threadstimed_waiting.count, 86 2025-06-29 02:59:42,262 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-2e802272]) (logid:a5ae4a07) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:59:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":168982216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42722247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":403624184,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":184095288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":504147968,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.06,0.14],"systemCyclesUsage":[66366470,53806983,4587805354],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 02:59:42,262 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1423, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:59:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":168982216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42722247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":403624184,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":184095288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":504147968,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.06,0.14],"systemCyclesUsage":[66366470,53806983,4587805354],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:59:42,262 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 02:59:42,266 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 02:59:42,266 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1423, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:59:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":168982216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42722247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":403624184,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":184095288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":504147968,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.06,0.14],"systemCyclesUsage":[66366470,53806983,4587805354],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:59:42,267 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-319:[ctx-414c17bd]) (logid:8b192f65) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 2:59:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":168982216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42722247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":403624184,"totalCommitted":577241088,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":184095288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":504147968,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.06,0.14],"systemCyclesUsage":[66366470,53806983,4587805354],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 02:59:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8944875e]) (logid:b19fb54c) No inactive management server node found 2025-06-29 02:59:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8944875e]) (logid:b19fb54c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3339df4a]) (logid:420f1ab9) No inactive management server node found 2025-06-29 02:59:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3339df4a]) (logid:420f1ab9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 02:59:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 02:59:45,794 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2c84db53]) (logid:a786796f) HA health check task is running... 2025-06-29 02:59:45,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e42f31b1]) (logid:073ca3bc) No inactive management server node found 2025-06-29 02:59:45,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e42f31b1]) (logid:073ca3bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-389695e6]) (logid:3f0bfd09) VM scheduler.poll is being called at 2025-06-29T01:00:00+0000 2025-06-29 02:59:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-389695e6]) (logid:3f0bfd09) Got 0 scheduled jobs to be executed at 2025-06-29T01:00:00+0000 2025-06-29 02:59:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-389695e6]) (logid:3f0bfd09) Cleaned up 0 VM scheduled job entries 2025-06-29 02:59:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-903d3cde]) (logid:05106b1a) VM Operation Thread Running 2025-06-29 02:59:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8ce0b89b]) (logid:f2ce8e57) ClusterDRS.poll is being called at 2025-06-29T01:00:00+0000 2025-06-29 02:59:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8ce0b89b]) (logid:f2ce8e57) Removed 0 old drs migration plans 2025-06-29 02:59:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c9dba9e0]) (logid:ba1ee7fb) Resetting hosts suitable for reconnect 2025-06-29 02:59:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c9dba9e0]) (logid:ba1ee7fb) Completed resetting hosts suitable for reconnect 2025-06-29 02:59:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c9dba9e0]) (logid:ba1ee7fb) Acquiring hosts for clusters already owned by this management server 2025-06-29 02:59:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c9dba9e0]) (logid:ba1ee7fb) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 02:59:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c9dba9e0]) (logid:ba1ee7fb) Acquiring hosts for clusters not owned by any management server 2025-06-29 02:59:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c9dba9e0]) (logid:ba1ee7fb) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 02:59:47,256 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:47,258 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:47,259 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 02:59:47,263 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 02:59:47,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25d0c730]) (logid:44e6727d) No inactive management server node found 2025-06-29 02:59:47,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25d0c730]) (logid:44e6727d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:48,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff12e94f]) (logid:7ee9772c) No inactive management server node found 2025-06-29 02:59:48,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff12e94f]) (logid:7ee9772c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:49,796 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-31ccd91e]) (logid:4abb0d0f) HA health check task is running... 2025-06-29 02:59:50,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7972: Seq 7-7972: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 02:59:50,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7972 2025-06-29 02:59:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ddc0067]) (logid:ff2419cf) No inactive management server node found 2025-06-29 02:59:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ddc0067]) (logid:ff2419cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d8cb782]) (logid:b65353ae) No inactive management server node found 2025-06-29 02:59:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d8cb782]) (logid:b65353ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:53,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1042bec2]) (logid:a8308467) No inactive management server node found 2025-06-29 02:59:53,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1042bec2]) (logid:a8308467) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 02:59:53,797 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cbb5f2fe]) (logid:199bc9d4) HA health check task is running... 2025-06-29 02:59:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18496039]) (logid:393200ac) No inactive management server node found 2025-06-29 02:59:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18496039]) (logid:393200ac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dde1c8c]) (logid:b8f5ff93) No inactive management server node found 2025-06-29 02:59:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dde1c8c]) (logid:b8f5ff93) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:56,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8ab94a33]) (logid:fb7b3464) Enabled non-edge zones available for scan: 1 2025-06-29 02:59:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8ab94a33]) (logid:fb7b3464) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 02:59:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8ab94a33]) (logid:fb7b3464) Zone [1] is ready to launch secondary storage VM. 2025-06-29 02:59:56,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-907a2690]) (logid:8855d2c1) Enabled non-edge zones available for scan: 1 2025-06-29 02:59:56,897 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-907a2690]) (logid:8855d2c1) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 02:59:57,798 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e2e7cc0c]) (logid:b4eb6b69) HA health check task is running... 2025-06-29 02:59:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e80641d0]) (logid:8159e614) No inactive management server node found 2025-06-29 02:59:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e80641d0]) (logid:8159e614) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 02:59:57,896 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-6fd3fdc2]) (logid:ee4151a5) AutoScaling Monitor is running... 2025-06-29 02:59:58,014 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-39093ed7]) (logid:9889d5e8) Starting to remove all volume_stats rows older than [Sat Jun 28 14:59:58 CEST 2025]. 2025-06-29 02:59:58,014 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-39093ed7]) (logid:9889d5e8) Removed a total of [0] volume_stats rows older than [Sat Jun 28 14:59:58 CEST 2025]. 2025-06-29 02:59:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8da8bad8]) (logid:823423f8) No inactive management server node found 2025-06-29 02:59:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8da8bad8]) (logid:823423f8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:00,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7973: Seq 7-7973: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:00,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7973 2025-06-29 03:00:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f36e56f4]) (logid:24eb3c6e) No inactive management server node found 2025-06-29 03:00:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f36e56f4]) (logid:24eb3c6e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:01,197 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:00:01,201 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:00:01,202 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) Seq 2-3048092522799696672: Routed from 90520738888109 2025-06-29 03:00:01,214 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) Seq 1-3048092522799696672: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:00:01,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:00:01,266 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) Seq 2-1612007191621929950: Routed from 90520738888109 2025-06-29 03:00:01,316 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0ac8d5f3]) (logid:d50348a7) Seq 2-1612007191621929950: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:00:01,800 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6a77b28d]) (logid:8c7bf002) HA health check task is running... 2025-06-29 03:00:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc9a01af]) (logid:c3f4cba1) No inactive management server node found 2025-06-29 03:00:02,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc9a01af]) (logid:c3f4cba1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:03,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3c41754]) (logid:3ebbdd00) No inactive management server node found 2025-06-29 03:00:03,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3c41754]) (logid:3ebbdd00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72705827]) (logid:0fc0fbc8) No inactive management server node found 2025-06-29 03:00:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72705827]) (logid:0fc0fbc8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:05,801 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e98d435d]) (logid:4fbe3cc9) HA health check task is running... 2025-06-29 03:00:06,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c82c7f1]) (logid:7f3c12e6) No inactive management server node found 2025-06-29 03:00:06,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c82c7f1]) (logid:7f3c12e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:08,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c813b0f]) (logid:b35f7ab3) No inactive management server node found 2025-06-29 03:00:08,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c813b0f]) (logid:b35f7ab3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:09,802 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b76d5b3a]) (logid:821f1267) HA health check task is running... 2025-06-29 03:00:09,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25e69417]) (logid:f6282462) No inactive management server node found 2025-06-29 03:00:09,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25e69417]) (logid:f6282462) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:10,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7974: Seq 7-7974: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:10,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7974 2025-06-29 03:00:10,166 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) StorageCollector is running... 2025-06-29 03:00:10,172 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:00:10,173 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:00:10,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:00:10,174 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-6717118844223095476: Routed from 90520738888109 2025-06-29 03:00:10,226 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 4-6717118844223095476: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:00:10,230 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:00:10,231 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:00:10,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:00:10,232 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-1612007191621929951: Routed from 90520738888109 2025-06-29 03:00:10,243 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-1612007191621929951: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:00:10,257 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:00:10,257 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:00:10,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:00:10,258 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-1612007191621929952: Routed from 90520738888109 2025-06-29 03:00:10,359 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-1612007191621929952: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:00:10,369 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:00:10,369 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:00:10,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:00:10,369 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-3048092522799696673: Routed from 90520738888109 2025-06-29 03:00:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f10b9b2f]) (logid:df0dea46) Found 0 routers to update status. 2025-06-29 03:00:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f10b9b2f]) (logid:df0dea46) Found 0 VPC's to update Redundant State. 2025-06-29 03:00:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f10b9b2f]) (logid:df0dea46) Found 0 networks to update RvR status. 2025-06-29 03:00:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0486fd6]) (logid:702f055c) No inactive management server node found 2025-06-29 03:00:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0486fd6]) (logid:702f055c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:11,422 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 1-3048092522799696673: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:00:11,424 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:00:11,424 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:00:11,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:00:11,425 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-1612007191621929953: Routed from 90520738888109 2025-06-29 03:00:12,484 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a507983a]) (logid:fe43038c) Seq 2-1612007191621929953: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:00:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f08737ad]) (logid:8440a00e) No inactive management server node found 2025-06-29 03:00:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f08737ad]) (logid:8440a00e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:13,271 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6543c1bc]) (logid:66c4b421) Starting to remove all vm_stats rows older than [Sat Jun 28 15:00:13 CEST 2025]. 2025-06-29 03:00:13,319 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6543c1bc]) (logid:66c4b421) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:00:13 CEST 2025]. 2025-06-29 03:00:13,803 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4f90d72d]) (logid:8b1543aa) HA health check task is running... 2025-06-29 03:00:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6eafee7]) (logid:e455de91) No inactive management server node found 2025-06-29 03:00:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6eafee7]) (logid:e455de91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:15,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b4ae52f]) (logid:053d301c) No inactive management server node found 2025-06-29 03:00:15,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b4ae52f]) (logid:053d301c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b27c2845]) (logid:0ebd5101) No inactive management server node found 2025-06-29 03:00:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b27c2845]) (logid:0ebd5101) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:17,804 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-bda7008d]) (logid:4a5a0346) HA health check task is running... 2025-06-29 03:00:18,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0431a53a]) (logid:a6e6e659) No inactive management server node found 2025-06-29 03:00:18,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0431a53a]) (logid:a6e6e659) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:20,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7975: Seq 7-7975: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:20,123 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7975 2025-06-29 03:00:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f5c56b7]) (logid:deb6bf3b) No inactive management server node found 2025-06-29 03:00:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f5c56b7]) (logid:deb6bf3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:21,806 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e2744a0f]) (logid:7aef5371) HA health check task is running... 2025-06-29 03:00:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c80e9ba]) (logid:f41d1bcf) No inactive management server node found 2025-06-29 03:00:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c80e9ba]) (logid:f41d1bcf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7139c737]) (logid:e2619e74) No inactive management server node found 2025-06-29 03:00:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7139c737]) (logid:e2619e74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac1b74eb]) (logid:0ec0b4a7) No inactive management server node found 2025-06-29 03:00:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac1b74eb]) (logid:0ec0b4a7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:25,807 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6d146597]) (logid:c37b1798) HA health check task is running... 2025-06-29 03:00:26,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8750f6d]) (logid:1a62d0b7) No inactive management server node found 2025-06-29 03:00:26,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8750f6d]) (logid:1a62d0b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f22a442c]) (logid:f3be32be) Enabled non-edge zones available for scan: 1 2025-06-29 03:00:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f22a442c]) (logid:f3be32be) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 03:00:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f22a442c]) (logid:f3be32be) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:00:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cf9c8d2d]) (logid:8822f181) Enabled non-edge zones available for scan: 1 2025-06-29 03:00:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cf9c8d2d]) (logid:8822f181) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:00:27,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3cb33f3]) (logid:7d722e81) No inactive management server node found 2025-06-29 03:00:27,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3cb33f3]) (logid:7d722e81) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce11ee10]) (logid:f4dc6aae) No inactive management server node found 2025-06-29 03:00:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce11ee10]) (logid:f4dc6aae) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:29,808 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c7b07fba]) (logid:2655ffb5) HA health check task is running... 2025-06-29 03:00:30,006 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:00:30,007 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:00:30,008 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) Seq 2-3048092522799696674: Routed from 90520738888109 2025-06-29 03:00:30,010 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) Seq 1-3048092522799696674: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:00:30,011 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:00:30,011 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) Seq 2-1612007191621929954: Routed from 90520738888109 2025-06-29 03:00:30,014 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dcfb58a4]) (logid:62819778) Seq 2-1612007191621929954: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:00:30,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7976: Seq 7-7976: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:30,111 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7976 2025-06-29 03:00:30,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-149ab474]) (logid:68356cf5) No inactive management server node found 2025-06-29 03:00:30,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-149ab474]) (logid:68356cf5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:32,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34f68c29]) (logid:45e8d49a) No inactive management server node found 2025-06-29 03:00:32,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34f68c29]) (logid:45e8d49a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:33,810 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6648ca75]) (logid:dcef20fc) HA health check task is running... 2025-06-29 03:00:33,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d5864b0]) (logid:04f71bbb) No inactive management server node found 2025-06-29 03:00:33,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d5864b0]) (logid:04f71bbb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:33,985 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:00:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:00:33,988 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:00:33,990 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:00:35,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4557428]) (logid:74a70201) No inactive management server node found 2025-06-29 03:00:35,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4557428]) (logid:74a70201) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:00:36,824 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e705f83]) (logid:6c2bfe1a) No inactive management server node found 2025-06-29 03:00:36,824 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e705f83]) (logid:6c2bfe1a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:37,811 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ecd25281]) (logid:0ab9c881) HA health check task is running... 2025-06-29 03:00:38,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b920da5c]) (logid:935cd46b) No inactive management server node found 2025-06-29 03:00:38,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b920da5c]) (logid:935cd46b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17db2647]) (logid:3def1711) No inactive management server node found 2025-06-29 03:00:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17db2647]) (logid:3def1711) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:40,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7977: Seq 7-7977: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:40,111 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7977 2025-06-29 03:00:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e7371ee1]) (logid:fd3860be) Found 0 routers to update status. 2025-06-29 03:00:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e7371ee1]) (logid:fd3860be) Found 0 VPC's to update Redundant State. 2025-06-29 03:00:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e7371ee1]) (logid:fd3860be) Found 0 networks to update RvR status. 2025-06-29 03:00:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-8decd951]) (logid:c7ae9397) DbCollector is running... 2025-06-29 03:00:41,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8101af82]) (logid:17182391) No inactive management server node found 2025-06-29 03:00:41,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8101af82]) (logid:17182391) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:41,812 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-67f00609]) (logid:bb5fb8b5) HA health check task is running... 2025-06-29 03:00:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) ManagementServerCollector is running... 2025-06-29 03:00:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) DB properties were already loaded 2025-06-29 03:00:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:00:42,170 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Execution of process [4062579] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:00:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Exit value of process [4062579] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:00:42,170 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Process [4062579] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:00:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) usage install: null 2025-06-29 03:00:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Sessions active: 0 2025-06-29 03:00:42,173 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:00:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062582] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:00:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:00:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062587] for command [/bin/bash -c uname -r ]. 2025-06-29 03:00:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:00:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062588] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:00:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) system memory from /proc: 4887515136 2025-06-29 03:00:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:00:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062593] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:00:42,214 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) free memory from /proc: 500834304 2025-06-29 03:00:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:00:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062598] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:00:42,232 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) used memory from /proc: 761820 2025-06-29 03:00:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:00:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062599] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:00:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:00:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062600] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:00:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:00:42,259 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062601] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:00:42,259 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:00:42,263 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062608] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:00:42,263 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:00:42,269 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Successfully executed process [4062609] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail buffersdirect.count, 120 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail buffersdirect.used, 657986 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail buffersmapped.count, 7 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail buffersmapped.used, 798602 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail gcPS-Scavenge.count, 494 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail gcPS-Scavenge.time, 3598 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail jvmuptime, 42782348 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memoryheap.committed, 352321536 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memoryheap.init, 77594624 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memoryheap.usage, 0.10956748872299635 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorynon-heap.committed, 224985088 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorynon-heap.max, -1 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorynon-heap.usage, -2.20496784E8 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorynon-heap.used, 220496784 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17591871334310852 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3024158946245876 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.Metaspace.usage, 0.9857033773361583 2025-06-29 03:00:42,271 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.PS-Eden-Space.usage, 0.040783967791863204 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.PS-Old-Gen.usage, 0.11975906690543757 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorypools.PS-Survivor-Space.usage, 0.9424414383737665 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorytotal.committed, 577306624 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorytotal.init, 85262336 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorytotal.max, 1908932607 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail memorytotal.used, 429716584 2025-06-29 03:00:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail threadsdeadlocks, [] 2025-06-29 03:00:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail threadsnew.count, 0 2025-06-29 03:00:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:00:42,275 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-58602ef8]) (logid:1d2dc8c4) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:00:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":143898184,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42782248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":428920136,"totalCommitted":577306624,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209156952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500834304,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.06,0.13],"systemCyclesUsage":[66366805,53807295,4587828441],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:00:42,275 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1425, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:00:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":143898184,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42782248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":428920136,"totalCommitted":577306624,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209156952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500834304,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.06,0.13],"systemCyclesUsage":[66366805,53807295,4587828441],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:00:42,276 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:00:42,280 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:00:42,281 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1425, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:00:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":143898184,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42782248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":428920136,"totalCommitted":577306624,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209156952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500834304,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.06,0.13],"systemCyclesUsage":[66366805,53807295,4587828441],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:00:42,281 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-320:[ctx-7b1e7a8c]) (logid:9b07299e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:00:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":352321536,"freeJvmMemoryBytes":143898184,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42782248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":428920136,"totalCommitted":577306624,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209156952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500834304,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.06,0.13],"systemCyclesUsage":[66366805,53807295,4587828441],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:00:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9328390e]) (logid:decaab5e) No inactive management server node found 2025-06-29 03:00:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9328390e]) (logid:decaab5e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:44,329 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9dec9b1]) (logid:57890094) No inactive management server node found 2025-06-29 03:00:44,329 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9dec9b1]) (logid:57890094) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:45,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-7978: Seq 7-7978: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:45,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 7978 2025-06-29 03:00:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:00:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:00:45,814 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b83fd82a]) (logid:175178fe) HA health check task is running... 2025-06-29 03:00:45,839 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-137ac8f2]) (logid:f0add9a6) No inactive management server node found 2025-06-29 03:00:45,839 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-137ac8f2]) (logid:f0add9a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6d1cfd13]) (logid:7fc94491) VM scheduler.poll is being called at 2025-06-29T01:01:00+0000 2025-06-29 03:00:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6d1cfd13]) (logid:7fc94491) Got 0 scheduled jobs to be executed at 2025-06-29T01:01:00+0000 2025-06-29 03:00:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6d1cfd13]) (logid:7fc94491) Cleaned up 0 VM scheduled job entries 2025-06-29 03:00:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-77230fdc]) (logid:cfba4572) VM Operation Thread Running 2025-06-29 03:00:46,330 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-286b535e]) (logid:e65a92f9) ClusterDRS.poll is being called at 2025-06-29T01:01:00+0000 2025-06-29 03:00:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-286b535e]) (logid:e65a92f9) Removed 0 old drs migration plans 2025-06-29 03:00:47,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:00:47,216 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:00:47,217 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:00:47,220 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:00:47,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a93143dc]) (logid:837debb3) No inactive management server node found 2025-06-29 03:00:47,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a93143dc]) (logid:837debb3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:00:48,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9f9c0c3]) (logid:eca8fecf) No inactive management server node found 2025-06-29 03:00:48,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9f9c0c3]) (logid:eca8fecf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:49,815 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5322c483]) (logid:11fb687b) HA health check task is running... 2025-06-29 03:00:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32ddba2b]) (logid:74231c76) No inactive management server node found 2025-06-29 03:00:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32ddba2b]) (logid:74231c76) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b33d1e3]) (logid:b4ecc4b5) No inactive management server node found 2025-06-29 03:00:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b33d1e3]) (logid:b4ecc4b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6358e845]) (logid:c591a127) No inactive management server node found 2025-06-29 03:00:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6358e845]) (logid:c591a127) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:53,817 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-211ff7b7]) (logid:5ab3a899) HA health check task is running... 2025-06-29 03:00:54,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ebbad35]) (logid:c350fdc9) No inactive management server node found 2025-06-29 03:00:54,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ebbad35]) (logid:c350fdc9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:55,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7980: Seq 7-7980: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:00:55,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7980 2025-06-29 03:00:56,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-785ac05b]) (logid:9073cb87) No inactive management server node found 2025-06-29 03:00:56,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-785ac05b]) (logid:9073cb87) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c0155d60]) (logid:661fd078) Enabled non-edge zones available for scan: 1 2025-06-29 03:00:56,766 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c0155d60]) (logid:661fd078) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2335 TB], threshold=[90.00%]. 2025-06-29 03:00:56,768 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c0155d60]) (logid:661fd078) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:00:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-060ac12b]) (logid:003e85de) Enabled non-edge zones available for scan: 1 2025-06-29 03:00:56,902 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-060ac12b]) (logid:003e85de) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:00:57,818 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e9228b82]) (logid:6b10b189) HA health check task is running... 2025-06-29 03:00:57,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5cb6c41]) (logid:47f11bc5) No inactive management server node found 2025-06-29 03:00:57,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5cb6c41]) (logid:47f11bc5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:00:57,898 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-1966c434]) (logid:1b1297aa) AutoScaling Monitor is running... 2025-06-29 03:00:58,016 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-fbd57693]) (logid:cacdd36b) Starting to remove all volume_stats rows older than [Sat Jun 28 15:00:58 CEST 2025]. 2025-06-29 03:00:58,017 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-fbd57693]) (logid:cacdd36b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:00:58 CEST 2025]. 2025-06-29 03:00:59,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-956d4bad]) (logid:c70e59e1) No inactive management server node found 2025-06-29 03:00:59,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-956d4bad]) (logid:c70e59e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:00,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccfbe0e3]) (logid:e7bc605d) No inactive management server node found 2025-06-29 03:01:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccfbe0e3]) (logid:e7bc605d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:01,375 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:01:01,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:01:01,387 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) Seq 2-3048092522799696675: Routed from 90520738888109 2025-06-29 03:01:01,446 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) Seq 1-3048092522799696675: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:01:01,485 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:01:01,486 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) Seq 2-1612007191621929955: Routed from 90520738888109 2025-06-29 03:01:01,531 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-41cd6b5b]) (logid:f547a10c) Seq 2-1612007191621929955: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:01:01,819 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-467a5fb0]) (logid:41f42895) HA health check task is running... 2025-06-29 03:01:02,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb70382d]) (logid:00d9f35c) No inactive management server node found 2025-06-29 03:01:02,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb70382d]) (logid:00d9f35c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e63be89b]) (logid:13a88746) No inactive management server node found 2025-06-29 03:01:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e63be89b]) (logid:13a88746) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:05,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-7981: Seq 7-7981: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:01:05,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 7981 2025-06-29 03:01:05,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bec428d]) (logid:a2396930) No inactive management server node found 2025-06-29 03:01:05,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bec428d]) (logid:a2396930) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:05,820 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-412056fd]) (logid:bfa99c49) HA health check task is running... 2025-06-29 03:01:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2975e1e9]) (logid:d6e0ea0f) No inactive management server node found 2025-06-29 03:01:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2975e1e9]) (logid:d6e0ea0f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c6d9d5f]) (logid:9ab3e250) No inactive management server node found 2025-06-29 03:01:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c6d9d5f]) (logid:9ab3e250) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:09,821 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-71cb571d]) (logid:33a5b891) HA health check task is running... 2025-06-29 03:01:09,824 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321e4c06]) (logid:5b250c96) No inactive management server node found 2025-06-29 03:01:09,824 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321e4c06]) (logid:5b250c96) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d32ee466]) (logid:0a2ab285) Found 0 routers to update status. 2025-06-29 03:01:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d32ee466]) (logid:0a2ab285) Found 0 VPC's to update Redundant State. 2025-06-29 03:01:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d32ee466]) (logid:0a2ab285) Found 0 networks to update RvR status. 2025-06-29 03:01:11,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4ed43ab]) (logid:aa577951) No inactive management server node found 2025-06-29 03:01:11,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4ed43ab]) (logid:aa577951) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:12,485 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) StorageCollector is running... 2025-06-29 03:01:12,496 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:01:12,496 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:01:12,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:01:12,497 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-6717118844223095477: Routed from 90520738888109 2025-06-29 03:01:12,575 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 4-6717118844223095477: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:01:12,579 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:01:12,579 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:01:12,579 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:01:12,580 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-1612007191621929956: Routed from 90520738888109 2025-06-29 03:01:12,593 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-1612007191621929956: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:01:12,604 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:01:12,604 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:01:12,604 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:01:12,605 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-1612007191621929957: Routed from 90520738888109 2025-06-29 03:01:12,697 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-1612007191621929957: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:01:12,708 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:01:12,708 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:01:12,708 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:01:12,709 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-3048092522799696676: Routed from 90520738888109 2025-06-29 03:01:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9e73106]) (logid:2fc1c689) No inactive management server node found 2025-06-29 03:01:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9e73106]) (logid:2fc1c689) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:13,321 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-183c5da8]) (logid:3738f946) Starting to remove all vm_stats rows older than [Sat Jun 28 15:01:13 CEST 2025]. 2025-06-29 03:01:13,348 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-183c5da8]) (logid:3738f946) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:01:13 CEST 2025]. 2025-06-29 03:01:13,761 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 1-3048092522799696676: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:01:13,763 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:01:13,763 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:01:13,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:01:13,764 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-1612007191621929958: Routed from 90520738888109 2025-06-29 03:01:13,822 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5a4e2f18]) (logid:33545f7c) HA health check task is running... 2025-06-29 03:01:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68be09eb]) (logid:d9fd7444) No inactive management server node found 2025-06-29 03:01:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68be09eb]) (logid:d9fd7444) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:14,822 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ecb49e34]) (logid:7c50ca27) Seq 2-1612007191621929958: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:01:15,104 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-7982: Seq 7-7982: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:01:15,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 7982 2025-06-29 03:01:15,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17f68d8f]) (logid:9ab8b19d) No inactive management server node found 2025-06-29 03:01:15,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17f68d8f]) (logid:9ab8b19d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4bf41db]) (logid:d829863c) Resetting hosts suitable for reconnect 2025-06-29 03:01:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4bf41db]) (logid:d829863c) Completed resetting hosts suitable for reconnect 2025-06-29 03:01:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4bf41db]) (logid:d829863c) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:01:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4bf41db]) (logid:d829863c) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:01:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4bf41db]) (logid:d829863c) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:01:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4bf41db]) (logid:d829863c) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:01:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4246cd24]) (logid:3988205f) No inactive management server node found 2025-06-29 03:01:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4246cd24]) (logid:3988205f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:17,824 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-80cb3539]) (logid:1563e035) HA health check task is running... 2025-06-29 03:01:18,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-777a7595]) (logid:50ad70f3) No inactive management server node found 2025-06-29 03:01:18,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-777a7595]) (logid:50ad70f3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:20,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4605901]) (logid:257ea554) No inactive management server node found 2025-06-29 03:01:20,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4605901]) (logid:257ea554) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:21,825 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6d9700ca]) (logid:c3b34243) HA health check task is running... 2025-06-29 03:01:21,837 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb35a684]) (logid:f5ccfece) No inactive management server node found 2025-06-29 03:01:21,837 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb35a684]) (logid:f5ccfece) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:23,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4676dca4]) (logid:0adef970) No inactive management server node found 2025-06-29 03:01:23,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4676dca4]) (logid:0adef970) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8989dd09]) (logid:0cd3d67d) No inactive management server node found 2025-06-29 03:01:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8989dd09]) (logid:0cd3d67d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:25,108 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-7983: Seq 7-7983: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:01:25,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 7983 2025-06-29 03:01:25,827 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-82bdc94e]) (logid:b3bc8350) HA health check task is running... 2025-06-29 03:01:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1339938c]) (logid:1fdb4958) No inactive management server node found 2025-06-29 03:01:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1339938c]) (logid:1fdb4958) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:26,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-982330be]) (logid:73cd5089) Enabled non-edge zones available for scan: 1 2025-06-29 03:01:26,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-982330be]) (logid:73cd5089) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2337 TB], threshold=[90.00%]. 2025-06-29 03:01:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-982330be]) (logid:73cd5089) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:01:26,896 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3d8387c4]) (logid:314bf047) Enabled non-edge zones available for scan: 1 2025-06-29 03:01:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3d8387c4]) (logid:314bf047) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:01:27,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef5e5d2d]) (logid:ed96d483) No inactive management server node found 2025-06-29 03:01:27,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef5e5d2d]) (logid:ed96d483) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:29,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a6dc8e8]) (logid:f75f7b42) No inactive management server node found 2025-06-29 03:01:29,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a6dc8e8]) (logid:f75f7b42) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:29,828 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-323833f6]) (logid:37b485d5) HA health check task is running... 2025-06-29 03:01:30,017 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:01:30,018 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:01:30,019 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) Seq 2-3048092522799696677: Routed from 90520738888109 2025-06-29 03:01:30,027 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) Seq 1-3048092522799696677: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:01:30,029 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:01:30,030 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) Seq 2-1612007191621929959: Routed from 90520738888109 2025-06-29 03:01:30,034 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-800431cc]) (logid:dcb2d0bc) Seq 2-1612007191621929959: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:01:30,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2528c411]) (logid:164ca8da) No inactive management server node found 2025-06-29 03:01:30,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2528c411]) (logid:164ca8da) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:32,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65b626a]) (logid:d1addb39) No inactive management server node found 2025-06-29 03:01:32,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65b626a]) (logid:d1addb39) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:01:33,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21a6c7a7]) (logid:0a186033) No inactive management server node found 2025-06-29 03:01:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21a6c7a7]) (logid:0a186033) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:33,829 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-051c7561]) (logid:cb978232) HA health check task is running... 2025-06-29 03:01:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:01:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:01:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:01:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:01:35,107 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7984: Seq 7-7984: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:01:35,128 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7984 2025-06-29 03:01:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06c5f79e]) (logid:ffcaddda) No inactive management server node found 2025-06-29 03:01:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06c5f79e]) (logid:ffcaddda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:36,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0920ed3]) (logid:d591c82a) No inactive management server node found 2025-06-29 03:01:36,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0920ed3]) (logid:d591c82a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:37,831 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-fb207ab5]) (logid:1ce4f0a7) HA health check task is running... 2025-06-29 03:01:38,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc84fa47]) (logid:fe27909d) No inactive management server node found 2025-06-29 03:01:38,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc84fa47]) (logid:fe27909d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:39,824 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0f039f0]) (logid:6411264d) No inactive management server node found 2025-06-29 03:01:39,824 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0f039f0]) (logid:6411264d) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:01:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c77992a]) (logid:31410a72) Found 0 routers to update status. 2025-06-29 03:01:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c77992a]) (logid:31410a72) Found 0 VPC's to update Redundant State. 2025-06-29 03:01:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c77992a]) (logid:31410a72) Found 0 networks to update RvR status. 2025-06-29 03:01:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-72f93d3e]) (logid:36d37154) DbCollector is running... 2025-06-29 03:01:41,342 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b6006f7]) (logid:b019f7b7) No inactive management server node found 2025-06-29 03:01:41,342 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b6006f7]) (logid:b019f7b7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:41,832 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b6d08616]) (logid:e6f5fc42) HA health check task is running... 2025-06-29 03:01:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) ManagementServerCollector is running... 2025-06-29 03:01:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) DB properties were already loaded 2025-06-29 03:01:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:01:42,170 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Execution of process [4062632] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:01:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Exit value of process [4062632] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:01:42,170 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Process [4062632] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:01:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) usage install: null 2025-06-29 03:01:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Sessions active: 0 2025-06-29 03:01:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:01:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062635] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:01:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:01:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062640] for command [/bin/bash -c uname -r ]. 2025-06-29 03:01:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:01:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062641] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:01:42,190 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) system memory from /proc: 4887515136 2025-06-29 03:01:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:01:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062646] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:01:42,196 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) free memory from /proc: 518914048 2025-06-29 03:01:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:01:42,215 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062651] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:01:42,215 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) used memory from /proc: 761820 2025-06-29 03:01:42,215 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:01:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062652] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:01:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:01:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062653] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:01:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:01:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062654] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:01:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:01:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062661] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:01:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:01:42,253 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Successfully executed process [4062662] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail buffersdirect.count, 120 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail buffersdirect.used, 657986 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail buffersmapped.count, 7 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail buffersmapped.used, 798602 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:01:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail gcPS-Scavenge.count, 495 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail gcPS-Scavenge.time, 3604 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail jvmuptime, 42842334 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memoryheap.committed, 353894400 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memoryheap.init, 77594624 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memoryheap.usage, 0.10615311150890037 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorynon-heap.max, -1 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorynon-heap.usage, -2.20543896E8 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorynon-heap.used, 220543896 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1760291005731805 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3026387592894991 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.Metaspace.usage, 0.9857448402647416 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.PS-Eden-Space.usage, 0.03496704332232025 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.PS-Old-Gen.usage, 0.11977050960042109 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorypools.PS-Survivor-Space.usage, 0.6294657389322916 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorytotal.committed, 578945024 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorytotal.init, 85262336 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorytotal.max, 1908932607 2025-06-29 03:01:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail memorytotal.used, 423238048 2025-06-29 03:01:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail threadsdeadlocks, [] 2025-06-29 03:01:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail threadsnew.count, 0 2025-06-29 03:01:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:01:42,262 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-125b9c00]) (logid:1bc5e086) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:01:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":152030608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42842247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":422426024,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202639136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":518914048,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.05,0.12],"systemCyclesUsage":[66367141,53807612,4587851521],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:01:42,263 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1427, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:01:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":152030608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42842247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":422426024,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202639136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":518914048,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.05,0.12],"systemCyclesUsage":[66367141,53807612,4587851521],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:01:42,263 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:01:42,268 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-320:[ctx-790a3654]) (logid:f78a8915) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:01:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":152030608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42842247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":422426024,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202639136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":518914048,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.05,0.12],"systemCyclesUsage":[66367141,53807612,4587851521],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:01:42,268 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 03:01:42,268 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1427, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:01:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":152030608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42842247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":422426024,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202639136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":518914048,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.05,0.12],"systemCyclesUsage":[66367141,53807612,4587851521],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:01:42,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e4ccce6]) (logid:098d4201) No inactive management server node found 2025-06-29 03:01:42,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e4ccce6]) (logid:098d4201) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8a7a487]) (logid:13ef93d7) No inactive management server node found 2025-06-29 03:01:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8a7a487]) (logid:13ef93d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:45,107 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-7985: Seq 7-7985: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:01:45,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 7985 2025-06-29 03:01:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:01:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:01:45,830 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b721f7e3]) (logid:de472348) No inactive management server node found 2025-06-29 03:01:45,830 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b721f7e3]) (logid:de472348) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:45,833 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7034c02a]) (logid:e657fdab) HA health check task is running... 2025-06-29 03:01:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-5c6e9fb9]) (logid:c856869a) VM scheduler.poll is being called at 2025-06-29T01:02:00+0000 2025-06-29 03:01:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-5c6e9fb9]) (logid:c856869a) Got 0 scheduled jobs to be executed at 2025-06-29T01:02:00+0000 2025-06-29 03:01:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-5c6e9fb9]) (logid:c856869a) Cleaned up 0 VM scheduled job entries 2025-06-29 03:01:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-c90e3495]) (logid:bfd542e1) VM Operation Thread Running 2025-06-29 03:01:46,331 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d0420663]) (logid:4d4dbbc5) ClusterDRS.poll is being called at 2025-06-29T01:02:00+0000 2025-06-29 03:01:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d0420663]) (logid:4d4dbbc5) Removed 0 old drs migration plans 2025-06-29 03:01:47,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:01:47,261 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:01:47,262 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:01:47,265 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:01:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0630c221]) (logid:9e2cc2ef) No inactive management server node found 2025-06-29 03:01:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0630c221]) (logid:9e2cc2ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:48,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c40b5371]) (logid:c15202c3) No inactive management server node found 2025-06-29 03:01:48,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c40b5371]) (logid:c15202c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:49,834 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c2c80043]) (logid:76cc164e) HA health check task is running... 2025-06-29 03:01:50,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7987: Seq 7-7987: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:01:50,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7987 2025-06-29 03:01:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-647d64fb]) (logid:9e65e898) No inactive management server node found 2025-06-29 03:01:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-647d64fb]) (logid:9e65e898) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:51,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-150b95cd]) (logid:1c43eaeb) No inactive management server node found 2025-06-29 03:01:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-150b95cd]) (logid:1c43eaeb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9f2257e]) (logid:8dc4a299) No inactive management server node found 2025-06-29 03:01:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9f2257e]) (logid:8dc4a299) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:53,836 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1fe07d4c]) (logid:d7bb3e07) HA health check task is running... 2025-06-29 03:01:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db51cf21]) (logid:679e93ff) No inactive management server node found 2025-06-29 03:01:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db51cf21]) (logid:679e93ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:56,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4584e8e]) (logid:605def95) No inactive management server node found 2025-06-29 03:01:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4584e8e]) (logid:605def95) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5019754f]) (logid:ebfcd3dd) Enabled non-edge zones available for scan: 1 2025-06-29 03:01:56,769 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5019754f]) (logid:ebfcd3dd) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2337 TB], threshold=[90.00%]. 2025-06-29 03:01:56,770 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5019754f]) (logid:ebfcd3dd) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:01:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0ef5172b]) (logid:f99a3c36) Enabled non-edge zones available for scan: 1 2025-06-29 03:01:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0ef5172b]) (logid:f99a3c36) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:01:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ea96b2d]) (logid:b6dc8e0d) No inactive management server node found 2025-06-29 03:01:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ea96b2d]) (logid:b6dc8e0d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:01:57,837 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5dc90770]) (logid:c7115958) HA health check task is running... 2025-06-29 03:01:57,901 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-212238fc]) (logid:224d85f2) AutoScaling Monitor is running... 2025-06-29 03:01:58,019 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-a61c9a0d]) (logid:61648d30) Starting to remove all volume_stats rows older than [Sat Jun 28 15:01:58 CEST 2025]. 2025-06-29 03:01:58,020 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-a61c9a0d]) (logid:61648d30) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:01:58 CEST 2025]. 2025-06-29 03:01:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-472310bc]) (logid:494504df) No inactive management server node found 2025-06-29 03:01:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-472310bc]) (logid:494504df) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:00,108 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7988: Seq 7-7988: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:00,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7988 2025-06-29 03:02:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df873b6c]) (logid:4d81d59a) No inactive management server node found 2025-06-29 03:02:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df873b6c]) (logid:4d81d59a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:01,586 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:02:01,590 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:02:01,591 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) Seq 2-3048092522799696678: Routed from 90520738888109 2025-06-29 03:02:01,643 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) Seq 1-3048092522799696678: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:02:01,677 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:02:01,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) Seq 2-1612007191621929960: Routed from 90520738888109 2025-06-29 03:02:01,722 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-0e0b4511]) (logid:0c271ebc) Seq 2-1612007191621929960: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:02:01,838 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2f527082]) (logid:a436d18d) HA health check task is running... 2025-06-29 03:02:02,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f123d49e]) (logid:ae5173fe) No inactive management server node found 2025-06-29 03:02:02,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f123d49e]) (logid:ae5173fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87b4f0d3]) (logid:91953607) No inactive management server node found 2025-06-29 03:02:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87b4f0d3]) (logid:91953607) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d616079]) (logid:47d7b722) No inactive management server node found 2025-06-29 03:02:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d616079]) (logid:47d7b722) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:05,839 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-18f6cce5]) (logid:4f1c1f8a) HA health check task is running... 2025-06-29 03:02:06,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28d148a4]) (logid:12de1bd5) No inactive management server node found 2025-06-29 03:02:06,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28d148a4]) (logid:12de1bd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:08,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55578090]) (logid:8eb66dd1) No inactive management server node found 2025-06-29 03:02:08,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55578090]) (logid:8eb66dd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd2d1abb]) (logid:c70ddc11) No inactive management server node found 2025-06-29 03:02:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd2d1abb]) (logid:c70ddc11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:09,840 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-917f11aa]) (logid:0c421413) HA health check task is running... 2025-06-29 03:02:10,108 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7989: Seq 7-7989: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:10,117 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7989 2025-06-29 03:02:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-67df784e]) (logid:0721425e) Found 0 routers to update status. 2025-06-29 03:02:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-67df784e]) (logid:0721425e) Found 0 VPC's to update Redundant State. 2025-06-29 03:02:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-67df784e]) (logid:0721425e) Found 0 networks to update RvR status. 2025-06-29 03:02:11,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9c0c01c]) (logid:475918f7) No inactive management server node found 2025-06-29 03:02:11,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9c0c01c]) (logid:475918f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e69df851]) (logid:a2bd3b29) No inactive management server node found 2025-06-29 03:02:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e69df851]) (logid:a2bd3b29) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:13,350 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-bf3622b1]) (logid:a0ca9d32) Starting to remove all vm_stats rows older than [Sat Jun 28 15:02:13 CEST 2025]. 2025-06-29 03:02:13,384 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-bf3622b1]) (logid:a0ca9d32) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:02:13 CEST 2025]. 2025-06-29 03:02:13,841 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-00cda80e]) (logid:f7a617cb) HA health check task is running... 2025-06-29 03:02:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6638b6bb]) (logid:7e2e3292) No inactive management server node found 2025-06-29 03:02:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6638b6bb]) (logid:7e2e3292) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:14,822 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) StorageCollector is running... 2025-06-29 03:02:14,828 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:02:14,828 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:02:14,828 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:02:14,829 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-6717118844223095478: Routed from 90520738888109 2025-06-29 03:02:14,883 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 4-6717118844223095478: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:02:14,887 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:02:14,887 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:02:14,887 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:02:14,888 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-1612007191621929961: Routed from 90520738888109 2025-06-29 03:02:14,899 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-1612007191621929961: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:02:14,910 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:02:14,910 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:02:14,910 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:02:14,911 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-1612007191621929962: Routed from 90520738888109 2025-06-29 03:02:15,006 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-1612007191621929962: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:02:15,018 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:02:15,018 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:02:15,018 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:02:15,020 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-3048092522799696679: Routed from 90520738888109 2025-06-29 03:02:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e2ed762]) (logid:a7d69051) No inactive management server node found 2025-06-29 03:02:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e2ed762]) (logid:a7d69051) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:16,085 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 1-3048092522799696679: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:02:16,087 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:02:16,087 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:02:16,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:02:16,088 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-1612007191621929963: Routed from 90520738888109 2025-06-29 03:02:17,163 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-61a11ab7]) (logid:d4b012ac) Seq 2-1612007191621929963: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:02:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2ec3b1a]) (logid:42be7276) No inactive management server node found 2025-06-29 03:02:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2ec3b1a]) (logid:42be7276) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:17,842 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ae6f761a]) (logid:db91d056) HA health check task is running... 2025-06-29 03:02:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd912376]) (logid:b5dabdf0) No inactive management server node found 2025-06-29 03:02:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd912376]) (logid:b5dabdf0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:20,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-7990: Seq 7-7990: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:20,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 7990 2025-06-29 03:02:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54a46756]) (logid:2170a778) No inactive management server node found 2025-06-29 03:02:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54a46756]) (logid:2170a778) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e3de739]) (logid:dbbf04e2) No inactive management server node found 2025-06-29 03:02:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e3de739]) (logid:dbbf04e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:21,844 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-78d23c13]) (logid:d05471c5) HA health check task is running... 2025-06-29 03:02:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a224588d]) (logid:d8602db6) No inactive management server node found 2025-06-29 03:02:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a224588d]) (logid:d8602db6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8edbf44c]) (logid:2651fddf) No inactive management server node found 2025-06-29 03:02:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8edbf44c]) (logid:2651fddf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:25,845 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dabc9e6d]) (logid:667a3d7a) HA health check task is running... 2025-06-29 03:02:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e38b3f02]) (logid:ec0f8ce6) No inactive management server node found 2025-06-29 03:02:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e38b3f02]) (logid:ec0f8ce6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-82e71419]) (logid:5eb1ff40) Enabled non-edge zones available for scan: 1 2025-06-29 03:02:26,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-82e71419]) (logid:5eb1ff40) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2350 TB], threshold=[90.00%]. 2025-06-29 03:02:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-82e71419]) (logid:5eb1ff40) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:02:26,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-11a335cb]) (logid:aca2754c) Enabled non-edge zones available for scan: 1 2025-06-29 03:02:26,897 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-11a335cb]) (logid:aca2754c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:02:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c48980a]) (logid:6f1025d8) No inactive management server node found 2025-06-29 03:02:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c48980a]) (logid:6f1025d8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0391a1fc]) (logid:c4d2a06c) No inactive management server node found 2025-06-29 03:02:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0391a1fc]) (logid:c4d2a06c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:29,846 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f025a440]) (logid:a31912da) HA health check task is running... 2025-06-29 03:02:30,037 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:02:30,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:02:30,039 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) Seq 2-3048092522799696680: Routed from 90520738888109 2025-06-29 03:02:30,046 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) Seq 1-3048092522799696680: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:02:30,046 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:02:30,047 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) Seq 2-1612007191621929964: Routed from 90520738888109 2025-06-29 03:02:30,050 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-987db5df]) (logid:755af7c0) Seq 2-1612007191621929964: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:02:30,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-7991: Seq 7-7991: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:30,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 7991 2025-06-29 03:02:30,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc5d2cea]) (logid:71b320b9) No inactive management server node found 2025-06-29 03:02:30,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc5d2cea]) (logid:71b320b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd4a18c0]) (logid:63a6ca62) No inactive management server node found 2025-06-29 03:02:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd4a18c0]) (logid:63a6ca62) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4091f6c]) (logid:f70dd8e9) No inactive management server node found 2025-06-29 03:02:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4091f6c]) (logid:f70dd8e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:33,848 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5e6d83a8]) (logid:69aed8a7) HA health check task is running... 2025-06-29 03:02:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:02:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:02:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:02:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:02:35,111 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7992: Seq 7-7992: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:35,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7992 2025-06-29 03:02:35,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e03e5e9]) (logid:e2eb9264) No inactive management server node found 2025-06-29 03:02:35,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e03e5e9]) (logid:e2eb9264) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77bbd57d]) (logid:54050481) No inactive management server node found 2025-06-29 03:02:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77bbd57d]) (logid:54050481) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:37,849 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2df1e2cb]) (logid:bbeedd53) HA health check task is running... 2025-06-29 03:02:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5d52bbf]) (logid:6146bbbd) No inactive management server node found 2025-06-29 03:02:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5d52bbf]) (logid:6146bbbd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f41b1954]) (logid:8da1075e) No inactive management server node found 2025-06-29 03:02:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f41b1954]) (logid:8da1075e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:02:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-80ff80cd]) (logid:732d1f24) Found 0 routers to update status. 2025-06-29 03:02:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-80ff80cd]) (logid:732d1f24) Found 0 VPC's to update Redundant State. 2025-06-29 03:02:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-80ff80cd]) (logid:732d1f24) Found 0 networks to update RvR status. 2025-06-29 03:02:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-2799a5f7]) (logid:1b7e0eaf) DbCollector is running... 2025-06-29 03:02:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43f81767]) (logid:9c182c1b) No inactive management server node found 2025-06-29 03:02:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43f81767]) (logid:9c182c1b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:41,850 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-0ddb4193]) (logid:a94ae968) HA health check task is running... 2025-06-29 03:02:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) ManagementServerCollector is running... 2025-06-29 03:02:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) DB properties were already loaded 2025-06-29 03:02:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:02:42,168 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Execution of process [4062675] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:02:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Exit value of process [4062675] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:02:42,168 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Process [4062675] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:02:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) usage install: null 2025-06-29 03:02:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Sessions active: 0 2025-06-29 03:02:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:02:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062678] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:02:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:02:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062683] for command [/bin/bash -c uname -r ]. 2025-06-29 03:02:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:02:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062684] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:02:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) system memory from /proc: 4887515136 2025-06-29 03:02:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:02:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062689] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:02:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) free memory from /proc: 510689280 2025-06-29 03:02:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:02:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062694] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:02:42,209 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) used memory from /proc: 761820 2025-06-29 03:02:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:02:42,215 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062695] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:02:42,215 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:02:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062696] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:02:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:02:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062697] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:02:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:02:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062704] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:02:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:02:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Successfully executed process [4062705] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail buffersdirect.count, 120 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail buffersdirect.used, 657986 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail buffersmapped.count, 7 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail buffersmapped.used, 798602 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail gcPS-Scavenge.count, 496 2025-06-29 03:02:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail gcPS-Scavenge.time, 3610 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail jvmuptime, 42902327 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memoryheap.committed, 353894400 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memoryheap.init, 77594624 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memoryheap.usage, 0.10339404082304827 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorynon-heap.max, -1 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorynon-heap.usage, -2.20551064E8 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorynon-heap.used, 220551064 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1760874183551053 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3026387592894991 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.Metaspace.usage, 0.9857448402647416 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.PS-Eden-Space.usage, 0.02530225550506661 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.PS-Old-Gen.usage, 0.11983916577032222 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorypools.PS-Survivor-Space.usage, 0.7514968145461309 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorytotal.committed, 578945024 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorytotal.init, 85262336 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorytotal.max, 1908932607 2025-06-29 03:02:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail memorytotal.used, 417985744 2025-06-29 03:02:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail threadsdeadlocks, [] 2025-06-29 03:02:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail threadsnew.count, 0 2025-06-29 03:02:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:02:42,253 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-6186c886]) (logid:f384829c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:02:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":157269032,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42902246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":417176432,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197372256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510689280,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.11],"systemCyclesUsage":[66367477,53807938,4587874599],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:02:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1429, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:02:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":157269032,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42902246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":417176432,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197372256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510689280,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.11],"systemCyclesUsage":[66367477,53807938,4587874599],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:02:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:02:42,257 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-320:[ctx-acc1c061]) (logid:901a2129) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:02:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":157269032,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42902246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":417176432,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197372256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510689280,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.11],"systemCyclesUsage":[66367477,53807938,4587874599],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:02:42,257 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:02:42,258 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1429, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:02:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":157269032,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42902246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":417176432,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197372256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":510689280,"systemMemoryUsed":761820,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.11],"systemCyclesUsage":[66367477,53807938,4587874599],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:02:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e6a4c6]) (logid:69924389) No inactive management server node found 2025-06-29 03:02:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e6a4c6]) (logid:69924389) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2793b0]) (logid:ca765a91) No inactive management server node found 2025-06-29 03:02:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2793b0]) (logid:ca765a91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:45,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-7993: Seq 7-7993: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:45,131 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 7993 2025-06-29 03:02:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:02:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:02:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5aea0c56]) (logid:02ca7625) No inactive management server node found 2025-06-29 03:02:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5aea0c56]) (logid:02ca7625) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:45,852 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a8e90936]) (logid:034b4647) HA health check task is running... 2025-06-29 03:02:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-52b47e68]) (logid:9d0782f1) VM scheduler.poll is being called at 2025-06-29T01:03:00+0000 2025-06-29 03:02:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-52b47e68]) (logid:9d0782f1) Got 0 scheduled jobs to be executed at 2025-06-29T01:03:00+0000 2025-06-29 03:02:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-52b47e68]) (logid:9d0782f1) Cleaned up 0 VM scheduled job entries 2025-06-29 03:02:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-61f7c52f]) (logid:6cca4efb) VM Operation Thread Running 2025-06-29 03:02:46,331 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9b03a10e]) (logid:f8fd24cb) ClusterDRS.poll is being called at 2025-06-29T01:03:00+0000 2025-06-29 03:02:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9b03a10e]) (logid:f8fd24cb) Removed 0 old drs migration plans 2025-06-29 03:02:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f90ace41]) (logid:7c316daa) Resetting hosts suitable for reconnect 2025-06-29 03:02:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f90ace41]) (logid:7c316daa) Completed resetting hosts suitable for reconnect 2025-06-29 03:02:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f90ace41]) (logid:7c316daa) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:02:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f90ace41]) (logid:7c316daa) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:02:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f90ace41]) (logid:7c316daa) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:02:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f90ace41]) (logid:7c316daa) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:02:47,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:02:47,229 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:02:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:02:47,233 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:02:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-978df711]) (logid:4a7fa57e) No inactive management server node found 2025-06-29 03:02:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-978df711]) (logid:4a7fa57e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-834ad120]) (logid:07f6282c) No inactive management server node found 2025-06-29 03:02:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-834ad120]) (logid:07f6282c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:49,853 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-906a2e21]) (logid:e90696f5) HA health check task is running... 2025-06-29 03:02:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fda89a2b]) (logid:e3570d1a) No inactive management server node found 2025-06-29 03:02:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fda89a2b]) (logid:e3570d1a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de54e5f4]) (logid:81f6eeab) No inactive management server node found 2025-06-29 03:02:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de54e5f4]) (logid:81f6eeab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17439427]) (logid:3161af26) No inactive management server node found 2025-06-29 03:02:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17439427]) (logid:3161af26) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:53,854 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4661f555]) (logid:c684e538) HA health check task is running... 2025-06-29 03:02:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb5ddcd]) (logid:eab10fb2) No inactive management server node found 2025-06-29 03:02:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb5ddcd]) (logid:eab10fb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:55,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-7995: Seq 7-7995: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:02:55,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 7995 2025-06-29 03:02:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c6cee70]) (logid:887d854a) No inactive management server node found 2025-06-29 03:02:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c6cee70]) (logid:887d854a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7f295da3]) (logid:bb54b90e) Enabled non-edge zones available for scan: 1 2025-06-29 03:02:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-7f295da3]) (logid:bb54b90e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2350 TB], threshold=[90.00%]. 2025-06-29 03:02:56,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7f295da3]) (logid:bb54b90e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:02:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-537be7e9]) (logid:85661e50) Enabled non-edge zones available for scan: 1 2025-06-29 03:02:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-537be7e9]) (logid:85661e50) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:02:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d3ad53a]) (logid:f0a6822c) No inactive management server node found 2025-06-29 03:02:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d3ad53a]) (logid:f0a6822c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:02:57,856 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e22e96bb]) (logid:fb44f43d) HA health check task is running... 2025-06-29 03:02:57,903 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-ed6eae41]) (logid:82ff6f9e) AutoScaling Monitor is running... 2025-06-29 03:02:58,022 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-7e20a573]) (logid:05bb4fda) Starting to remove all volume_stats rows older than [Sat Jun 28 15:02:58 CEST 2025]. 2025-06-29 03:02:58,023 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-7e20a573]) (logid:05bb4fda) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:02:58 CEST 2025]. 2025-06-29 03:02:59,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2272082e]) (logid:9ce8b1a6) No inactive management server node found 2025-06-29 03:02:59,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2272082e]) (logid:9ce8b1a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c44e806]) (logid:db7e68c8) No inactive management server node found 2025-06-29 03:03:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c44e806]) (logid:db7e68c8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:01,775 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:03:01,780 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:03:01,781 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) Seq 2-3048092522799696681: Routed from 90520738888109 2025-06-29 03:03:01,833 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) Seq 1-3048092522799696681: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:03:01,857 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d85bc56f]) (logid:ca82b4e6) HA health check task is running... 2025-06-29 03:03:01,878 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:03:01,878 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) Seq 2-1612007191621929965: Routed from 90520738888109 2025-06-29 03:03:01,929 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-aa03bb58]) (logid:b23635c7) Seq 2-1612007191621929965: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:03:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72800141]) (logid:dab08f51) No inactive management server node found 2025-06-29 03:03:02,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72800141]) (logid:dab08f51) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-882b3318]) (logid:83b58a18) No inactive management server node found 2025-06-29 03:03:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-882b3318]) (logid:83b58a18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:05,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-7996: Seq 7-7996: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:03:05,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 7996 2025-06-29 03:03:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df17aa16]) (logid:cbe0bfd0) No inactive management server node found 2025-06-29 03:03:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df17aa16]) (logid:cbe0bfd0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:03:05,858 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0e9c29a3]) (logid:a9322dc5) HA health check task is running... 2025-06-29 03:03:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b0e3845]) (logid:f943096e) No inactive management server node found 2025-06-29 03:03:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b0e3845]) (logid:f943096e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:08,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16ff5eba]) (logid:75d27107) No inactive management server node found 2025-06-29 03:03:08,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16ff5eba]) (logid:75d27107) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5d01acc]) (logid:18874720) No inactive management server node found 2025-06-29 03:03:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5d01acc]) (logid:18874720) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:09,859 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8b8b7af0]) (logid:3afafefc) HA health check task is running... 2025-06-29 03:03:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9e188872]) (logid:9a0650b2) Found 0 routers to update status. 2025-06-29 03:03:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9e188872]) (logid:9a0650b2) Found 0 VPC's to update Redundant State. 2025-06-29 03:03:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9e188872]) (logid:9a0650b2) Found 0 networks to update RvR status. 2025-06-29 03:03:11,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66aad0e2]) (logid:ae1fd35b) No inactive management server node found 2025-06-29 03:03:11,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66aad0e2]) (logid:ae1fd35b) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:12,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e38e26a]) (logid:e49702a0) No inactive management server node found 2025-06-29 03:03:12,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e38e26a]) (logid:e49702a0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:13,386 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-a96ec0b6]) (logid:2b1b8ee3) Starting to remove all vm_stats rows older than [Sat Jun 28 15:03:13 CEST 2025]. 2025-06-29 03:03:13,410 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-a96ec0b6]) (logid:2b1b8ee3) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:03:13 CEST 2025]. 2025-06-29 03:03:13,860 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cd94750d]) (logid:a4ca6976) HA health check task is running... 2025-06-29 03:03:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50c98284]) (logid:8de9d4a4) No inactive management server node found 2025-06-29 03:03:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50c98284]) (logid:8de9d4a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:15,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-7997: Seq 7-7997: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:03:15,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 7997 2025-06-29 03:03:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a38c112]) (logid:f9942394) No inactive management server node found 2025-06-29 03:03:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a38c112]) (logid:f9942394) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:17,163 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) StorageCollector is running... 2025-06-29 03:03:17,168 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:03:17,168 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:03:17,168 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:03:17,169 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 2-6717118844223095479: Routed from 90520738888109 2025-06-29 03:03:17,240 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 4-6717118844223095479: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:03:17,244 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:03:17,244 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:03:17,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:03:17,245 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 2-3048092522799696682: Routed from 90520738888109 2025-06-29 03:03:17,292 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 1-3048092522799696682: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:03:17,302 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:03:17,302 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:03:17,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:03:17,303 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 2-3048092522799696683: Routed from 90520738888109 2025-06-29 03:03:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e673b7ef]) (logid:eba7cd4c) No inactive management server node found 2025-06-29 03:03:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e673b7ef]) (logid:eba7cd4c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:03:17,393 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 1-3048092522799696683: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:03:17,404 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:03:17,404 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:03:17,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:03:17,405 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 2-3048092522799696684: Routed from 90520738888109 2025-06-29 03:03:17,862 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cfa10b6a]) (logid:8cd0c7f5) HA health check task is running... 2025-06-29 03:03:18,496 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 1-3048092522799696684: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:03:18,498 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:03:18,498 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:03:18,498 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:03:18,499 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 2-1612007191621929966: Routed from 90520738888109 2025-06-29 03:03:18,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c6b82b2]) (logid:9ca1763f) No inactive management server node found 2025-06-29 03:03:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c6b82b2]) (logid:9ca1763f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:19,518 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2099be6d]) (logid:6b1e58a6) Seq 2-1612007191621929966: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:03:20,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-470c5a6e]) (logid:4537b80a) No inactive management server node found 2025-06-29 03:03:20,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-470c5a6e]) (logid:4537b80a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8c670ae]) (logid:4097b765) No inactive management server node found 2025-06-29 03:03:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8c670ae]) (logid:4097b765) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:21,863 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3ce3de1b]) (logid:713d8d30) HA health check task is running... 2025-06-29 03:03:23,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1748760]) (logid:f7edeb72) No inactive management server node found 2025-06-29 03:03:23,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1748760]) (logid:f7edeb72) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:24,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5ec6933]) (logid:25e54e29) No inactive management server node found 2025-06-29 03:03:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5ec6933]) (logid:25e54e29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:25,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-7998: Seq 7-7998: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:03:25,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 7998 2025-06-29 03:03:25,864 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a9cf48fc]) (logid:c4d76b76) HA health check task is running... 2025-06-29 03:03:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2981aff2]) (logid:eed8ee58) No inactive management server node found 2025-06-29 03:03:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2981aff2]) (logid:eed8ee58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7aaea060]) (logid:6135726f) Enabled non-edge zones available for scan: 1 2025-06-29 03:03:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-7aaea060]) (logid:6135726f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2355 TB], threshold=[90.00%]. 2025-06-29 03:03:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7aaea060]) (logid:6135726f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:03:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f249b5ea]) (logid:be54f304) Enabled non-edge zones available for scan: 1 2025-06-29 03:03:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f249b5ea]) (logid:be54f304) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:03:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d468446c]) (logid:2a81279a) No inactive management server node found 2025-06-29 03:03:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d468446c]) (logid:2a81279a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8435ce4]) (logid:7fa6380f) No inactive management server node found 2025-06-29 03:03:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8435ce4]) (logid:7fa6380f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:29,866 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4ef9b5ad]) (logid:8778e685) HA health check task is running... 2025-06-29 03:03:30,053 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:03:30,053 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:03:30,054 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) Seq 2-3048092522799696685: Routed from 90520738888109 2025-06-29 03:03:30,057 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) Seq 1-3048092522799696685: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:03:30,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:03:30,058 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) Seq 2-1612007191621929967: Routed from 90520738888109 2025-06-29 03:03:30,061 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-03dbc80f]) (logid:7e8b5657) Seq 2-1612007191621929967: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:03:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d0890d3]) (logid:cf7057bb) No inactive management server node found 2025-06-29 03:03:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d0890d3]) (logid:cf7057bb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:32,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a211c97e]) (logid:6d39354c) No inactive management server node found 2025-06-29 03:03:32,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a211c97e]) (logid:6d39354c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:33,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1c6ee8a]) (logid:625d50a5) No inactive management server node found 2025-06-29 03:03:33,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1c6ee8a]) (logid:625d50a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:33,867 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8a8cea9a]) (logid:fb6dcdaa) HA health check task is running... 2025-06-29 03:03:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:03:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:03:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:03:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:03:35,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-7999: Seq 7-7999: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:03:35,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 7999 2025-06-29 03:03:35,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4322b9b1]) (logid:62a417ec) No inactive management server node found 2025-06-29 03:03:35,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4322b9b1]) (logid:62a417ec) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:03:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1aa0595a]) (logid:490d1bb4) No inactive management server node found 2025-06-29 03:03:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1aa0595a]) (logid:490d1bb4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:37,868 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ca3f1a63]) (logid:50dc0d80) HA health check task is running... 2025-06-29 03:03:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa9bcfb5]) (logid:1a32dadf) No inactive management server node found 2025-06-29 03:03:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa9bcfb5]) (logid:1a32dadf) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:03:39,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd44d2fa]) (logid:a1a3546d) No inactive management server node found 2025-06-29 03:03:39,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd44d2fa]) (logid:a1a3546d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:40,817 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-397b4c28]) (logid:5b6c4eca) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:03:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-397b4c28]) (logid:5b6c4eca) Snapshot scheduler is being called at [2025-06-29T01:03:40+0000]. 2025-06-29 03:03:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-397b4c28]) (logid:5b6c4eca) There are [0] scheduled snapshots to be executed at [2025-06-29T01:03:40+0000]. 2025-06-29 03:03:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-3cc97ac5]) (logid:b91aa72f) External devices are not used. Skipping external device usage collection 2025-06-29 03:03:40,979 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-7e9f7917]) (logid:95579590) Backup backup.poll is being called at 2025-06-29T01:03:40+0000 2025-06-29 03:03:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-152e4acc]) (logid:e6d23c59) Found 0 running routers. 2025-06-29 03:03:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-96b22d6f]) (logid:9ee1e3bc) Found 0 routers to update status. 2025-06-29 03:03:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-96b22d6f]) (logid:9ee1e3bc) Found 0 VPC's to update Redundant State. 2025-06-29 03:03:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-96b22d6f]) (logid:9ee1e3bc) Found 0 networks to update RvR status. 2025-06-29 03:03:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-b58936e0]) (logid:994ea012) DbCollector is running... 2025-06-29 03:03:41,281 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-5fb45be5]) (logid:ec0f4ef7) Started resource counters recalculation periodic task. 2025-06-29 03:03:41,284 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-5fb45be5]) (logid:ec0f4ef7) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:03:41,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6f61282]) (logid:0ccd1500) No inactive management server node found 2025-06-29 03:03:41,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6f61282]) (logid:0ccd1500) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:41,648 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-5fb45be5]) (logid:ec0f4ef7) Finished resource counters recalculation periodic task. 2025-06-29 03:03:41,870 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7d67b855]) (logid:9ad7cdd8) HA health check task is running... 2025-06-29 03:03:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) ManagementServerCollector is running... 2025-06-29 03:03:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) DB properties were already loaded 2025-06-29 03:03:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:03:42,170 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Execution of process [4062710] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:03:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Exit value of process [4062710] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:03:42,170 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Process [4062710] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:03:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) usage install: null 2025-06-29 03:03:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Sessions active: 0 2025-06-29 03:03:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:03:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062713] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:03:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:03:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062718] for command [/bin/bash -c uname -r ]. 2025-06-29 03:03:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:03:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062719] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:03:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) system memory from /proc: 4887515136 2025-06-29 03:03:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:03:42,198 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062724] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:03:42,198 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) free memory from /proc: 500113408 2025-06-29 03:03:42,198 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:03:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062729] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:03:42,216 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) used memory from /proc: 762716 2025-06-29 03:03:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:03:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062730] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:03:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:03:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062731] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:03:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:03:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062732] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:03:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:03:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062739] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:03:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:03:42,257 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Successfully executed process [4062740] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:03:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:03:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail buffersdirect.count, 120 2025-06-29 03:03:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail buffersdirect.used, 657986 2025-06-29 03:03:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail buffersmapped.count, 7 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail buffersmapped.used, 798602 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail gcPS-Scavenge.count, 497 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail gcPS-Scavenge.time, 3615 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail jvmuptime, 42962337 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memoryheap.committed, 353894400 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memoryheap.init, 77594624 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memoryheap.usage, 0.10556820662785807 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorynon-heap.max, -1 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorynon-heap.usage, -2.20551832E8 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorynon-heap.used, 220551832 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17609366668888296 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3026387592894991 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.Metaspace.usage, 0.9857448402647416 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.PS-Eden-Space.usage, 0.03130118965922253 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.PS-Old-Gen.usage, 0.1199021005927316 2025-06-29 03:03:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorypools.PS-Survivor-Space.usage, 0.7440556117466518 2025-06-29 03:03:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorytotal.committed, 578945024 2025-06-29 03:03:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorytotal.init, 85262336 2025-06-29 03:03:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorytotal.max, 1908932607 2025-06-29 03:03:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail memorytotal.used, 422129568 2025-06-29 03:03:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail threadsdeadlocks, [] 2025-06-29 03:03:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail threadsnew.count, 0 2025-06-29 03:03:42,263 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:03:42,265 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-fbfe65b9]) (logid:404782ac) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:03:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":153126880,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42962247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":421319352,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201522592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500113408,"systemMemoryUsed":762716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.1],"systemCyclesUsage":[66367834,53808270,4587897651],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:03:42,265 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1431, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:03:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":153126880,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42962247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":421319352,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201522592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500113408,"systemMemoryUsed":762716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.1],"systemCyclesUsage":[66367834,53808270,4587897651],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:03:42,265 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:03:42,269 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:03:42,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1431, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:03:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":153126880,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42962247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":421319352,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201522592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500113408,"systemMemoryUsed":762716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.1],"systemCyclesUsage":[66367834,53808270,4587897651],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:03:42,270 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-321:[ctx-60c69d8b]) (logid:ab0cc1b9) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:03:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353894400,"freeJvmMemoryBytes":153126880,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":42962247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":421319352,"totalCommitted":578945024,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201522592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500113408,"systemMemoryUsed":762716,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.1],"systemCyclesUsage":[66367834,53808270,4587897651],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:03:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f13a1934]) (logid:9ef61646) No inactive management server node found 2025-06-29 03:03:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f13a1934]) (logid:9ef61646) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc846a43]) (logid:4fecf820) No inactive management server node found 2025-06-29 03:03:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc846a43]) (logid:4fecf820) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:03:45,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8000: Seq 7-8000: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:03:45,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8000 2025-06-29 03:03:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:03:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:03:45,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-822f341a]) (logid:c87f7b33) No inactive management server node found 2025-06-29 03:03:45,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-822f341a]) (logid:c87f7b33) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:45,871 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-960165da]) (logid:8e851b06) HA health check task is running... 2025-06-29 03:03:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-20109da4]) (logid:65f3adf2) VM scheduler.poll is being called at 2025-06-29T01:04:00+0000 2025-06-29 03:03:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-20109da4]) (logid:65f3adf2) Got 0 scheduled jobs to be executed at 2025-06-29T01:04:00+0000 2025-06-29 03:03:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-20109da4]) (logid:65f3adf2) Cleaned up 0 VM scheduled job entries 2025-06-29 03:03:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-cb10e4e7]) (logid:194a0336) VM Operation Thread Running 2025-06-29 03:03:46,183 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-4:[ctx-4edc1c50]) (logid:2e30d01c) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:03:46,331 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-b787c319]) (logid:9552394a) ClusterDRS.poll is being called at 2025-06-29T01:04:00+0000 2025-06-29 03:03:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-b787c319]) (logid:9552394a) Removed 0 old drs migration plans 2025-06-29 03:03:47,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:03:47,247 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:03:47,248 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:03:47,251 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:03:47,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b2085c8]) (logid:efed0e68) No inactive management server node found 2025-06-29 03:03:47,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b2085c8]) (logid:efed0e68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cb8fa0a]) (logid:fc8320a9) No inactive management server node found 2025-06-29 03:03:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cb8fa0a]) (logid:fc8320a9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:49,872 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b99d1caf]) (logid:08fea985) HA health check task is running... 2025-06-29 03:03:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fa811be]) (logid:e6f6c6b7) No inactive management server node found 2025-06-29 03:03:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fa811be]) (logid:e6f6c6b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:51,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f417d314]) (logid:f9513cc0) No inactive management server node found 2025-06-29 03:03:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f417d314]) (logid:f9513cc0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b17c01]) (logid:2cde798d) No inactive management server node found 2025-06-29 03:03:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b17c01]) (logid:2cde798d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:53,873 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8fe26f98]) (logid:96fcf8ba) HA health check task is running... 2025-06-29 03:03:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-804f0d28]) (logid:7af0eb35) No inactive management server node found 2025-06-29 03:03:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-804f0d28]) (logid:7af0eb35) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:55,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8002: Seq 7-8002: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:03:55,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8002 2025-06-29 03:03:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f2556eb]) (logid:f37d6673) No inactive management server node found 2025-06-29 03:03:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f2556eb]) (logid:f37d6673) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d4f94cbd]) (logid:8143ffae) Enabled non-edge zones available for scan: 1 2025-06-29 03:03:56,765 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d4f94cbd]) (logid:8143ffae) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2355 TB], threshold=[90.00%]. 2025-06-29 03:03:56,766 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d4f94cbd]) (logid:8143ffae) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:03:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-474a6673]) (logid:f562f0ce) Enabled non-edge zones available for scan: 1 2025-06-29 03:03:56,907 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-474a6673]) (logid:f562f0ce) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:03:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-738eb3fb]) (logid:882210d4) No inactive management server node found 2025-06-29 03:03:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-738eb3fb]) (logid:882210d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:03:57,875 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3f6f16dc]) (logid:e6db7161) HA health check task is running... 2025-06-29 03:03:57,905 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-47d7d56d]) (logid:7d598622) AutoScaling Monitor is running... 2025-06-29 03:03:58,025 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-b9cb2ca9]) (logid:ba4a5361) Starting to remove all volume_stats rows older than [Sat Jun 28 15:03:58 CEST 2025]. 2025-06-29 03:03:58,026 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-b9cb2ca9]) (logid:ba4a5361) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:03:58 CEST 2025]. 2025-06-29 03:03:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e6842ad]) (logid:932ef38f) No inactive management server node found 2025-06-29 03:03:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e6842ad]) (logid:932ef38f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:00,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a4877fc]) (logid:b16acb19) No inactive management server node found 2025-06-29 03:04:00,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a4877fc]) (logid:b16acb19) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:01,876 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1097bceb]) (logid:4ca7e9fe) HA health check task is running... 2025-06-29 03:04:02,015 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:04:02,020 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:04:02,022 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) Seq 2-3048092522799696686: Routed from 90520738888109 2025-06-29 03:04:02,077 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) Seq 1-3048092522799696686: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:04:02,123 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:04:02,124 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) Seq 2-1612007191621929968: Routed from 90520738888109 2025-06-29 03:04:02,172 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-e7014bc0]) (logid:6d9c6a8f) Seq 2-1612007191621929968: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:04:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dd4e177]) (logid:d8b5e6ef) No inactive management server node found 2025-06-29 03:04:02,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dd4e177]) (logid:d8b5e6ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9480491]) (logid:8dcb19d9) No inactive management server node found 2025-06-29 03:04:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9480491]) (logid:8dcb19d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:05,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8003: Seq 7-8003: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:04:05,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8003 2025-06-29 03:04:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed30aebc]) (logid:455ad02f) No inactive management server node found 2025-06-29 03:04:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed30aebc]) (logid:455ad02f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:05,877 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-75d2f230]) (logid:75abedf1) HA health check task is running... 2025-06-29 03:04:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71fb1335]) (logid:92be9b30) No inactive management server node found 2025-06-29 03:04:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71fb1335]) (logid:92be9b30) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0345f783]) (logid:87fa2095) No inactive management server node found 2025-06-29 03:04:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0345f783]) (logid:87fa2095) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d0992f1]) (logid:2037e0e0) No inactive management server node found 2025-06-29 03:04:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d0992f1]) (logid:2037e0e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:09,878 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7ff7a878]) (logid:09c49f6e) HA health check task is running... 2025-06-29 03:04:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-50ec12af]) (logid:48378be2) Found 0 routers to update status. 2025-06-29 03:04:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-50ec12af]) (logid:48378be2) Found 0 VPC's to update Redundant State. 2025-06-29 03:04:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-50ec12af]) (logid:48378be2) Found 0 networks to update RvR status. 2025-06-29 03:04:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Running Capacity Checker ... 2025-06-29 03:04:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) recalculating system capacity 2025-06-29 03:04:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Executing cpu/ram capacity update 2025-06-29 03:04:11,071 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:11,085 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:04:11,086 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:04:11,086 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:04:11,097 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:11,103 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:11,104 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:04:11,104 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:04:11,104 DEBUG [c.c.c.CapacityManagerImpl] (pool-314-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:04:11,112 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Done executing cpu/ram capacity update 2025-06-29 03:04:11,112 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Executing storage capacity update 2025-06-29 03:04:11,120 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:04:11,120 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:04:11,120 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:04:11,123 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:04:11,128 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:04:11,129 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:04:11,129 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:04:11,131 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:04:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:04:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:04:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:04:11,138 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:04:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:04:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:04:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:04:11,147 DEBUG [c.c.s.StorageManagerImpl] (pool-315-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:04:11,147 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Done executing storage capacity update 2025-06-29 03:04:11,147 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Executing capacity updates for public ip and Vlans 2025-06-29 03:04:11,168 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Done capacity updates for public ip and Vlans 2025-06-29 03:04:11,168 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Executing capacity updates for private ip 2025-06-29 03:04:11,172 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Done executing capacity updates for private ip 2025-06-29 03:04:11,172 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Done recalculating system capacity 2025-06-29 03:04:11,201 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-97046838]) (logid:79a62c12) Done running Capacity Checker ... 2025-06-29 03:04:11,258 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-fdc49f5f]) (logid:63700a7c) Checking if any host reservation can be released ... 2025-06-29 03:04:11,265 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-fdc49f5f]) (logid:63700a7c) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:11,269 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-fdc49f5f]) (logid:63700a7c) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:11,269 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-fdc49f5f]) (logid:63700a7c) Done running HostReservationReleaseChecker ... 2025-06-29 03:04:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9ada7a7]) (logid:f4d7f3a4) No inactive management server node found 2025-06-29 03:04:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9ada7a7]) (logid:f4d7f3a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23188968]) (logid:4a98f8f5) No inactive management server node found 2025-06-29 03:04:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23188968]) (logid:4a98f8f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:13,412 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-030a7591]) (logid:1da88226) Starting to remove all vm_stats rows older than [Sat Jun 28 15:04:13 CEST 2025]. 2025-06-29 03:04:13,445 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-030a7591]) (logid:1da88226) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:04:13 CEST 2025]. 2025-06-29 03:04:13,880 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d5485154]) (logid:edcdfcef) HA health check task is running... 2025-06-29 03:04:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2ed82a1]) (logid:9c2d1abf) No inactive management server node found 2025-06-29 03:04:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2ed82a1]) (logid:9c2d1abf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:15,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8004: Seq 7-8004: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:04:15,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8004 2025-06-29 03:04:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68cca42f]) (logid:c1287c12) No inactive management server node found 2025-06-29 03:04:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68cca42f]) (logid:c1287c12) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-82718668]) (logid:9338d8dd) Resetting hosts suitable for reconnect 2025-06-29 03:04:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-82718668]) (logid:9338d8dd) Completed resetting hosts suitable for reconnect 2025-06-29 03:04:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-82718668]) (logid:9338d8dd) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:04:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-82718668]) (logid:9338d8dd) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:04:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-82718668]) (logid:9338d8dd) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:04:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-82718668]) (logid:9338d8dd) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:04:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-777f9f06]) (logid:6d4ffd8c) No inactive management server node found 2025-06-29 03:04:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-777f9f06]) (logid:6d4ffd8c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:17,882 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3b16746c]) (logid:a9d5c7b2) HA health check task is running... 2025-06-29 03:04:18,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bab422ca]) (logid:4c9990ea) No inactive management server node found 2025-06-29 03:04:18,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bab422ca]) (logid:4c9990ea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:19,518 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) StorageCollector is running... 2025-06-29 03:04:19,526 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:04:19,526 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:04:19,526 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:04:19,528 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 2-6717118844223095480: Routed from 90520738888109 2025-06-29 03:04:19,581 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 4-6717118844223095480: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:04:19,586 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:04:19,586 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:04:19,586 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:04:19,587 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 2-3048092522799696687: Routed from 90520738888109 2025-06-29 03:04:19,634 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 1-3048092522799696687: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:04:19,656 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:04:19,656 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:04:19,656 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:04:19,657 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 2-3048092522799696688: Routed from 90520738888109 2025-06-29 03:04:19,744 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 1-3048092522799696688: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:04:19,755 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:04:19,755 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:04:19,755 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:04:19,756 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 2-3048092522799696689: Routed from 90520738888109 2025-06-29 03:04:20,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7aa5a044]) (logid:5b235796) No inactive management server node found 2025-06-29 03:04:20,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7aa5a044]) (logid:5b235796) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:20,847 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 1-3048092522799696689: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:04:20,849 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:04:20,849 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:04:20,849 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:04:20,849 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 2-1612007191621929969: Routed from 90520738888109 2025-06-29 03:04:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7016d4f9]) (logid:77e8a21c) No inactive management server node found 2025-06-29 03:04:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7016d4f9]) (logid:77e8a21c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:21,868 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f4927afd]) (logid:54d2b74a) Seq 2-1612007191621929969: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:04:21,883 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a2a223ff]) (logid:4db402b2) HA health check task is running... 2025-06-29 03:04:23,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e08d2d1e]) (logid:506bac2f) No inactive management server node found 2025-06-29 03:04:23,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e08d2d1e]) (logid:506bac2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:24,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81d17d40]) (logid:02c1a4ac) No inactive management server node found 2025-06-29 03:04:24,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81d17d40]) (logid:02c1a4ac) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:04:25,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8005: Seq 7-8005: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:04:25,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8005 2025-06-29 03:04:25,884 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c80d7864]) (logid:b4334180) HA health check task is running... 2025-06-29 03:04:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0144f59d]) (logid:f0cc6c33) No inactive management server node found 2025-06-29 03:04:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0144f59d]) (logid:f0cc6c33) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:26,759 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-cfdf6ff1]) (logid:2463e956) Enabled non-edge zones available for scan: 1 2025-06-29 03:04:26,766 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-cfdf6ff1]) (logid:2463e956) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:04:26,768 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-cfdf6ff1]) (logid:2463e956) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:04:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8b1c0416]) (logid:91f8496b) Enabled non-edge zones available for scan: 1 2025-06-29 03:04:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8b1c0416]) (logid:91f8496b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:04:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29e15b99]) (logid:75e0860f) No inactive management server node found 2025-06-29 03:04:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29e15b99]) (logid:75e0860f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f194942]) (logid:16cc8862) No inactive management server node found 2025-06-29 03:04:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f194942]) (logid:16cc8862) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:04:29,885 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-2faa46bf]) (logid:e588d140) HA health check task is running... 2025-06-29 03:04:30,063 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:04:30,064 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:04:30,064 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) Seq 2-3048092522799696690: Routed from 90520738888109 2025-06-29 03:04:30,067 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) Seq 1-3048092522799696690: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:04:30,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:04:30,068 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) Seq 2-1612007191621929970: Routed from 90520738888109 2025-06-29 03:04:30,085 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-635bcfaa]) (logid:ccea1cf4) Seq 2-1612007191621929970: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:04:30,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8006: Seq 7-8006: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:04:30,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8006 2025-06-29 03:04:30,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ab28736]) (logid:237b91b1) No inactive management server node found 2025-06-29 03:04:30,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ab28736]) (logid:237b91b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9017895]) (logid:eb711dcb) No inactive management server node found 2025-06-29 03:04:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9017895]) (logid:eb711dcb) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99b7387a]) (logid:40d60725) No inactive management server node found 2025-06-29 03:04:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99b7387a]) (logid:40d60725) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:33,886 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-77a556b7]) (logid:0b8b8f3b) HA health check task is running... 2025-06-29 03:04:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:04:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:04:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4ca1040]) (logid:a68d5045) No inactive management server node found 2025-06-29 03:04:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4ca1040]) (logid:a68d5045) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:36,852 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-debf0454]) (logid:3ef6faee) No inactive management server node found 2025-06-29 03:04:36,852 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-debf0454]) (logid:3ef6faee) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:37,887 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-740e7b1c]) (logid:54fdb214) HA health check task is running... 2025-06-29 03:04:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e43f03ee]) (logid:6cfd72a3) No inactive management server node found 2025-06-29 03:04:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e43f03ee]) (logid:6cfd72a3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:04:39,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8b71e68]) (logid:d060c02d) No inactive management server node found 2025-06-29 03:04:39,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8b71e68]) (logid:d060c02d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:40,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8007: Seq 7-8007: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:04:40,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8007 2025-06-29 03:04:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-33e365f1]) (logid:f30d87ce) Found 0 routers to update status. 2025-06-29 03:04:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-33e365f1]) (logid:f30d87ce) Found 0 VPC's to update Redundant State. 2025-06-29 03:04:41,005 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-33e365f1]) (logid:f30d87ce) Found 0 networks to update RvR status. 2025-06-29 03:04:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-1a5c70eb]) (logid:efa826b3) DbCollector is running... 2025-06-29 03:04:41,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c8f663c]) (logid:6b28d4bb) No inactive management server node found 2025-06-29 03:04:41,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c8f663c]) (logid:6b28d4bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:41,889 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-780a6ad5]) (logid:e04c6d9a) HA health check task is running... 2025-06-29 03:04:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) ManagementServerCollector is running... 2025-06-29 03:04:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) DB properties were already loaded 2025-06-29 03:04:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:04:42,170 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Execution of process [4062760] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:04:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Exit value of process [4062760] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:04:42,171 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Process [4062760] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:04:42,171 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) usage install: null 2025-06-29 03:04:42,171 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Sessions active: 0 2025-06-29 03:04:42,173 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:04:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062763] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:04:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:04:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062768] for command [/bin/bash -c uname -r ]. 2025-06-29 03:04:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:04:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062769] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:04:42,191 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) system memory from /proc: 4887515136 2025-06-29 03:04:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:04:42,197 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062774] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:04:42,197 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) free memory from /proc: 497020928 2025-06-29 03:04:42,197 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:04:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062779] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:04:42,213 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) used memory from /proc: 762844 2025-06-29 03:04:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:04:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062780] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:04:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:04:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062781] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:04:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:04:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062782] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:04:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:04:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062789] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:04:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:04:42,254 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Successfully executed process [4062790] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail buffersdirect.count, 120 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail buffersdirect.used, 657986 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail buffersmapped.count, 7 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail buffersmapped.used, 798602 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail gcPS-Scavenge.count, 498 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail gcPS-Scavenge.time, 3622 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:04:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail jvmuptime, 43022335 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memoryheap.committed, 354418688 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memoryheap.init, 77594624 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memoryheap.usage, 0.10763528012404301 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorynon-heap.max, -1 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorynon-heap.usage, -2.20594384E8 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorynon-heap.used, 220594384 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17611241169021594 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3029157779851368 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.Metaspace.usage, 0.9857866267012202 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.PS-Eden-Space.usage, 0.03729076068087533 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.PS-Old-Gen.usage, 0.12039413647702307 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorypools.PS-Survivor-Space.usage, 0.6306922219016335 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorytotal.committed, 579469312 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorytotal.init, 85262336 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorytotal.max, 1908932607 2025-06-29 03:04:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail memorytotal.used, 426117872 2025-06-29 03:04:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail threadsdeadlocks, [] 2025-06-29 03:04:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail threadsnew.count, 0 2025-06-29 03:04:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:04:42,261 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-e3eb736b]) (logid:94416f2d) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:04:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":149722944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43022248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":425308464,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205468496,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497020928,"systemMemoryUsed":762844,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.09],"systemCyclesUsage":[66368186,53808602,4587920703],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:04:42,262 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1433, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:04:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":149722944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43022248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":425308464,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205468496,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497020928,"systemMemoryUsed":762844,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.09],"systemCyclesUsage":[66368186,53808602,4587920703],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:04:42,262 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:04:42,267 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-321:[ctx-043b16a7]) (logid:0e01b2a2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:04:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":149722944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43022248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":425308464,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205468496,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497020928,"systemMemoryUsed":762844,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.09],"systemCyclesUsage":[66368186,53808602,4587920703],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:04:42,267 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 03:04:42,267 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1433, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:04:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":149722944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43022248,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":425308464,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205468496,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497020928,"systemMemoryUsed":762844,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.09],"systemCyclesUsage":[66368186,53808602,4587920703],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:04:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15e546f6]) (logid:ad379395) No inactive management server node found 2025-06-29 03:04:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15e546f6]) (logid:ad379395) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:04:44,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20e5d669]) (logid:6637c349) No inactive management server node found 2025-06-29 03:04:44,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20e5d669]) (logid:6637c349) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:04:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:04:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07bd82d6]) (logid:117dfd66) No inactive management server node found 2025-06-29 03:04:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07bd82d6]) (logid:117dfd66) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:45,890 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-603d7e2f]) (logid:2ac5a5b8) HA health check task is running... 2025-06-29 03:04:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c371a15c]) (logid:31e642c1) VM scheduler.poll is being called at 2025-06-29T01:05:00+0000 2025-06-29 03:04:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c371a15c]) (logid:31e642c1) Got 0 scheduled jobs to be executed at 2025-06-29T01:05:00+0000 2025-06-29 03:04:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c371a15c]) (logid:31e642c1) Cleaned up 0 VM scheduled job entries 2025-06-29 03:04:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-31c87245]) (logid:085655f4) VM Operation Thread Running 2025-06-29 03:04:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2f54072f]) (logid:a0a28fae) ClusterDRS.poll is being called at 2025-06-29T01:05:00+0000 2025-06-29 03:04:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2f54072f]) (logid:a0a28fae) Removed 0 old drs migration plans 2025-06-29 03:04:47,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:47,217 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:47,219 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:04:47,223 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:04:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8df5061]) (logid:8e7f81c0) No inactive management server node found 2025-06-29 03:04:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8df5061]) (logid:8e7f81c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15601253]) (logid:ed85cf6e) No inactive management server node found 2025-06-29 03:04:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15601253]) (logid:ed85cf6e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:49,899 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-072c5bde]) (logid:7fb7fe81) HA health check task is running... 2025-06-29 03:04:49,899 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:04:50,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8009: Seq 7-8009: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:04:50,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8009 2025-06-29 03:04:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be7f4113]) (logid:8c361bce) No inactive management server node found 2025-06-29 03:04:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be7f4113]) (logid:8c361bce) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52ff4fc7]) (logid:41dd89a9) No inactive management server node found 2025-06-29 03:04:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52ff4fc7]) (logid:41dd89a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:53,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1abd3a9b]) (logid:f92958d7) No inactive management server node found 2025-06-29 03:04:53,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1abd3a9b]) (logid:f92958d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:53,900 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cdb974f3]) (logid:99b0382b) HA health check task is running... 2025-06-29 03:04:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05d4cab4]) (logid:f956f03d) No inactive management server node found 2025-06-29 03:04:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05d4cab4]) (logid:f956f03d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:56,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fff233eb]) (logid:2aae9022) No inactive management server node found 2025-06-29 03:04:56,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fff233eb]) (logid:2aae9022) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5eb9ad65]) (logid:8e459f3f) Enabled non-edge zones available for scan: 1 2025-06-29 03:04:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5eb9ad65]) (logid:8e459f3f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:04:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5eb9ad65]) (logid:8e459f3f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:04:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9cfa44ff]) (logid:5ac0bb97) Enabled non-edge zones available for scan: 1 2025-06-29 03:04:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9cfa44ff]) (logid:5ac0bb97) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:04:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d4a3df8]) (logid:960c4a46) No inactive management server node found 2025-06-29 03:04:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d4a3df8]) (logid:960c4a46) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:04:57,901 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-92745f7b]) (logid:c9f422d4) HA health check task is running... 2025-06-29 03:04:57,907 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-698eb856]) (logid:23dc1223) AutoScaling Monitor is running... 2025-06-29 03:04:58,027 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-6291f4db]) (logid:87e555c2) Starting to remove all volume_stats rows older than [Sat Jun 28 15:04:58 CEST 2025]. 2025-06-29 03:04:58,028 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-6291f4db]) (logid:87e555c2) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:04:58 CEST 2025]. 2025-06-29 03:04:59,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd55105d]) (logid:586b670b) No inactive management server node found 2025-06-29 03:04:59,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd55105d]) (logid:586b670b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:00,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8010: Seq 7-8010: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:00,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8010 2025-06-29 03:05:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65b1fb0c]) (logid:d002c331) No inactive management server node found 2025-06-29 03:05:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65b1fb0c]) (logid:d002c331) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:01,902 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a96f8173]) (logid:08892c4b) HA health check task is running... 2025-06-29 03:05:02,233 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:05:02,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:05:02,240 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) Seq 2-3048092522799696691: Routed from 90520738888109 2025-06-29 03:05:02,294 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) Seq 1-3048092522799696691: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:05:02,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86a34216]) (logid:917d00d5) No inactive management server node found 2025-06-29 03:05:02,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86a34216]) (logid:917d00d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:02,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:05:02,330 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) Seq 2-1612007191621929971: Routed from 90520738888109 2025-06-29 03:05:02,390 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-53bfb7bf]) (logid:dceb6f0f) Seq 2-1612007191621929971: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:05:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ea4a8ae]) (logid:a00860a8) No inactive management server node found 2025-06-29 03:05:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ea4a8ae]) (logid:a00860a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:05,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8a064d3]) (logid:0a1be2e6) No inactive management server node found 2025-06-29 03:05:05,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8a064d3]) (logid:0a1be2e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:05,904 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-720e1838]) (logid:afa45e7a) HA health check task is running... 2025-06-29 03:05:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f958c97f]) (logid:0fb42eef) No inactive management server node found 2025-06-29 03:05:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f958c97f]) (logid:0fb42eef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:08,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19abd5a6]) (logid:7c3cbf86) No inactive management server node found 2025-06-29 03:05:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19abd5a6]) (logid:7c3cbf86) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:09,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fd19073]) (logid:1add06ae) No inactive management server node found 2025-06-29 03:05:09,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fd19073]) (logid:1add06ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:09,905 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8de49298]) (logid:7ea9f52e) HA health check task is running... 2025-06-29 03:05:10,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8011: Seq 7-8011: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:10,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8011 2025-06-29 03:05:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-635fe52c]) (logid:2624dd27) Found 0 routers to update status. 2025-06-29 03:05:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-635fe52c]) (logid:2624dd27) Found 0 VPC's to update Redundant State. 2025-06-29 03:05:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-635fe52c]) (logid:2624dd27) Found 0 networks to update RvR status. 2025-06-29 03:05:11,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6961aae0]) (logid:78f6dcd9) No inactive management server node found 2025-06-29 03:05:11,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6961aae0]) (logid:78f6dcd9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:05:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46c4eb92]) (logid:a1eb286a) No inactive management server node found 2025-06-29 03:05:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46c4eb92]) (logid:a1eb286a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:13,446 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-a0ad0bff]) (logid:deeac3e5) Starting to remove all vm_stats rows older than [Sat Jun 28 15:05:13 CEST 2025]. 2025-06-29 03:05:13,479 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-a0ad0bff]) (logid:deeac3e5) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:05:13 CEST 2025]. 2025-06-29 03:05:13,907 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c1b9b640]) (logid:2f43867c) HA health check task is running... 2025-06-29 03:05:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10a6477c]) (logid:e869f546) No inactive management server node found 2025-06-29 03:05:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10a6477c]) (logid:e869f546) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3edaf8ae]) (logid:5d28e2c7) No inactive management server node found 2025-06-29 03:05:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3edaf8ae]) (logid:5d28e2c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60106ce5]) (logid:866014a4) No inactive management server node found 2025-06-29 03:05:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60106ce5]) (logid:866014a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:17,908 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ecbe5335]) (logid:e632c1fb) HA health check task is running... 2025-06-29 03:05:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca978518]) (logid:f40b0fa9) No inactive management server node found 2025-06-29 03:05:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca978518]) (logid:f40b0fa9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:20,117 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8012: Seq 7-8012: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:20,139 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8012 2025-06-29 03:05:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-989492fc]) (logid:4dd7eabb) No inactive management server node found 2025-06-29 03:05:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-989492fc]) (logid:4dd7eabb) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-418627d7]) (logid:578a6acf) No inactive management server node found 2025-06-29 03:05:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-418627d7]) (logid:578a6acf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:21,868 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) StorageCollector is running... 2025-06-29 03:05:21,873 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:05:21,873 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:05:21,873 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:05:21,874 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 2-6717118844223095481: Routed from 90520738888109 2025-06-29 03:05:21,909 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9f1cd680]) (logid:65abec02) HA health check task is running... 2025-06-29 03:05:21,929 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 4-6717118844223095481: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:05:21,932 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:05:21,932 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:05:21,932 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:05:21,933 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 2-3048092522799696692: Routed from 90520738888109 2025-06-29 03:05:21,980 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 1-3048092522799696692: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:05:21,991 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:05:21,991 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:05:21,991 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:05:21,992 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 2-3048092522799696693: Routed from 90520738888109 2025-06-29 03:05:22,080 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 1-3048092522799696693: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:05:22,089 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:05:22,089 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:05:22,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:05:22,090 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 2-3048092522799696694: Routed from 90520738888109 2025-06-29 03:05:23,181 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 1-3048092522799696694: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:05:23,184 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:05:23,184 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:05:23,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:05:23,185 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 2-1612007191621929972: Routed from 90520738888109 2025-06-29 03:05:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcc6eb49]) (logid:a5260d84) No inactive management server node found 2025-06-29 03:05:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcc6eb49]) (logid:a5260d84) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:24,209 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1b3b5755]) (logid:ed13871d) Seq 2-1612007191621929972: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:05:24,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-350b1c9c]) (logid:00d5eacd) No inactive management server node found 2025-06-29 03:05:24,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-350b1c9c]) (logid:00d5eacd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:25,910 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5a280a2e]) (logid:6c5775be) HA health check task is running... 2025-06-29 03:05:26,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6798f22c]) (logid:0ad3a987) No inactive management server node found 2025-06-29 03:05:26,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6798f22c]) (logid:0ad3a987) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-468c3393]) (logid:c2f299a3) Enabled non-edge zones available for scan: 1 2025-06-29 03:05:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-468c3393]) (logid:c2f299a3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:05:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-468c3393]) (logid:c2f299a3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:05:26,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-16579889]) (logid:e002d772) Enabled non-edge zones available for scan: 1 2025-06-29 03:05:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-16579889]) (logid:e002d772) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:05:27,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5eadff60]) (logid:854edb13) No inactive management server node found 2025-06-29 03:05:27,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5eadff60]) (logid:854edb13) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18ee61c6]) (logid:5651fa70) No inactive management server node found 2025-06-29 03:05:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18ee61c6]) (logid:5651fa70) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:29,911 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-924d1ec7]) (logid:898c331a) HA health check task is running... 2025-06-29 03:05:30,088 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:05:30,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:05:30,090 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) Seq 2-3048092522799696695: Routed from 90520738888109 2025-06-29 03:05:30,092 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) Seq 1-3048092522799696695: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:05:30,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:05:30,093 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) Seq 2-1612007191621929973: Routed from 90520738888109 2025-06-29 03:05:30,096 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-86f1e51a]) (logid:16a82f08) Seq 2-1612007191621929973: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:05:30,117 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8013: Seq 7-8013: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:30,138 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8013 2025-06-29 03:05:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a21fa643]) (logid:1b6762ef) No inactive management server node found 2025-06-29 03:05:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a21fa643]) (logid:1b6762ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ff83bad]) (logid:710be3da) No inactive management server node found 2025-06-29 03:05:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ff83bad]) (logid:710be3da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b73245ce]) (logid:9cf5b163) No inactive management server node found 2025-06-29 03:05:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b73245ce]) (logid:9cf5b163) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:33,913 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-da325d26]) (logid:a2b18bed) HA health check task is running... 2025-06-29 03:05:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:05:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:05:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:05:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:05:35,117 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8014: Seq 7-8014: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:35,138 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8014 2025-06-29 03:05:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96654c70]) (logid:68463a1c) No inactive management server node found 2025-06-29 03:05:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96654c70]) (logid:68463a1c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-124658e9]) (logid:809596e3) No inactive management server node found 2025-06-29 03:05:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-124658e9]) (logid:809596e3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:37,914 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d1ce33c8]) (logid:c2353fde) HA health check task is running... 2025-06-29 03:05:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-148bc874]) (logid:d264f32f) No inactive management server node found 2025-06-29 03:05:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-148bc874]) (logid:d264f32f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aff079af]) (logid:5bca199c) No inactive management server node found 2025-06-29 03:05:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aff079af]) (logid:5bca199c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fc2aad6a]) (logid:f4b36115) Found 0 routers to update status. 2025-06-29 03:05:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fc2aad6a]) (logid:f4b36115) Found 0 VPC's to update Redundant State. 2025-06-29 03:05:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fc2aad6a]) (logid:f4b36115) Found 0 networks to update RvR status. 2025-06-29 03:05:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-9e5e226f]) (logid:5a1346f2) DbCollector is running... 2025-06-29 03:05:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a14f3959]) (logid:4af0619d) No inactive management server node found 2025-06-29 03:05:41,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a14f3959]) (logid:4af0619d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:41,915 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a49e514c]) (logid:3d53f5ea) HA health check task is running... 2025-06-29 03:05:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) ManagementServerCollector is running... 2025-06-29 03:05:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) DB properties were already loaded 2025-06-29 03:05:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:05:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Execution of process [4062796] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:05:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Exit value of process [4062796] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:05:42,170 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Process [4062796] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:05:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) usage install: null 2025-06-29 03:05:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Sessions active: 0 2025-06-29 03:05:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:05:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062799] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:05:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:05:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062804] for command [/bin/bash -c uname -r ]. 2025-06-29 03:05:42,184 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:05:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062805] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:05:42,190 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) system memory from /proc: 4887515136 2025-06-29 03:05:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:05:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062810] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:05:42,196 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) free memory from /proc: 490708992 2025-06-29 03:05:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:05:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062815] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:05:42,212 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) used memory from /proc: 764508 2025-06-29 03:05:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:05:42,219 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062816] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:05:42,219 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:05:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062817] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:05:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:05:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062818] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:05:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:05:42,245 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062825] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:05:42,245 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:05:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Successfully executed process [4062826] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail buffersdirect.count, 120 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail buffersdirect.used, 657986 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail buffersmapped.count, 7 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail buffersmapped.used, 798602 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail gcPS-Scavenge.count, 499 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail gcPS-Scavenge.time, 3629 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail jvmuptime, 43082330 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memoryheap.committed, 353370112 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memoryheap.init, 77594624 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memoryheap.usage, 0.10572732382179519 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorynon-heap.max, -1 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorynon-heap.usage, -2.20594384E8 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorynon-heap.used, 220594384 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17611241169021594 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3029157779851368 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:05:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.Metaspace.usage, 0.9857866267012202 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.PS-Eden-Space.usage, 0.027992312065315102 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.PS-Old-Gen.usage, 0.12054289151180886 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorypools.PS-Survivor-Space.usage, 0.8973257882254464 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorytotal.committed, 578420736 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorytotal.init, 85262336 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorytotal.max, 1908932607 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail memorytotal.used, 422482856 2025-06-29 03:05:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail threadsdeadlocks, [] 2025-06-29 03:05:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail threadsnew.count, 0 2025-06-29 03:05:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:05:42,257 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-f319860e]) (logid:e141b36b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:05:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353370112,"freeJvmMemoryBytes":152297224,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43082247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":421667272,"totalCommitted":578420736,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201826336,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490708992,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368514,53808932,4587943787],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:05:42,257 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1435, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:05:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353370112,"freeJvmMemoryBytes":152297224,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43082247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":421667272,"totalCommitted":578420736,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201826336,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490708992,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368514,53808932,4587943787],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:05:42,258 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:05:42,261 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-321:[ctx-39e821fe]) (logid:c9b3a8bf) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:05:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353370112,"freeJvmMemoryBytes":152297224,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43082247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":421667272,"totalCommitted":578420736,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201826336,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490708992,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368514,53808932,4587943787],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:05:42,261 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:05:42,261 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1435, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:05:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":353370112,"freeJvmMemoryBytes":152297224,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43082247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":421667272,"totalCommitted":578420736,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201826336,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490708992,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368514,53808932,4587943787],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:05:42,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8868f794]) (logid:eea88347) No inactive management server node found 2025-06-29 03:05:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8868f794]) (logid:eea88347) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6080c5b8]) (logid:46fec0b3) No inactive management server node found 2025-06-29 03:05:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6080c5b8]) (logid:46fec0b3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:05:45,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8015: Seq 7-8015: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:45,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8015 2025-06-29 03:05:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:05:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:05:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91a074c2]) (logid:7cdc5c3a) No inactive management server node found 2025-06-29 03:05:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91a074c2]) (logid:7cdc5c3a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:45,916 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ddd07963]) (logid:07de909c) HA health check task is running... 2025-06-29 03:05:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a96b5d57]) (logid:33e239f6) VM scheduler.poll is being called at 2025-06-29T01:06:00+0000 2025-06-29 03:05:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a96b5d57]) (logid:33e239f6) Got 0 scheduled jobs to be executed at 2025-06-29T01:06:00+0000 2025-06-29 03:05:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a96b5d57]) (logid:33e239f6) Cleaned up 0 VM scheduled job entries 2025-06-29 03:05:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-35ae7c5e]) (logid:70411543) VM Operation Thread Running 2025-06-29 03:05:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2eab35dd]) (logid:2de3c029) ClusterDRS.poll is being called at 2025-06-29T01:06:00+0000 2025-06-29 03:05:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2eab35dd]) (logid:2de3c029) Removed 0 old drs migration plans 2025-06-29 03:05:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-508e400b]) (logid:84d55e1d) Resetting hosts suitable for reconnect 2025-06-29 03:05:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-508e400b]) (logid:84d55e1d) Completed resetting hosts suitable for reconnect 2025-06-29 03:05:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-508e400b]) (logid:84d55e1d) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:05:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-508e400b]) (logid:84d55e1d) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:05:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-508e400b]) (logid:84d55e1d) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:05:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-508e400b]) (logid:84d55e1d) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:05:47,218 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:05:47,220 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:05:47,221 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:05:47,223 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:05:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbe01081]) (logid:eae4b8ac) No inactive management server node found 2025-06-29 03:05:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbe01081]) (logid:eae4b8ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:48,829 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83a89816]) (logid:0a8365c5) No inactive management server node found 2025-06-29 03:05:48,829 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83a89816]) (logid:0a8365c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:49,917 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-dd2adca3]) (logid:6a2cc168) HA health check task is running... 2025-06-29 03:05:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-807ff72e]) (logid:ee396bcd) No inactive management server node found 2025-06-29 03:05:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-807ff72e]) (logid:ee396bcd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d6df267]) (logid:1335deb3) No inactive management server node found 2025-06-29 03:05:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d6df267]) (logid:1335deb3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67b9d524]) (logid:99e68e47) No inactive management server node found 2025-06-29 03:05:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67b9d524]) (logid:99e68e47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:53,919 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c7a54a04]) (logid:e40b6627) HA health check task is running... 2025-06-29 03:05:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c28cfc9]) (logid:c69aa7df) No inactive management server node found 2025-06-29 03:05:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c28cfc9]) (logid:c69aa7df) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:55,118 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8017: Seq 7-8017: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:05:55,127 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8017 2025-06-29 03:05:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd72d356]) (logid:2fc8949f) No inactive management server node found 2025-06-29 03:05:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd72d356]) (logid:2fc8949f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c56f4bb]) (logid:822514e3) Enabled non-edge zones available for scan: 1 2025-06-29 03:05:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1c56f4bb]) (logid:822514e3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:05:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c56f4bb]) (logid:822514e3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:05:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1ff6262c]) (logid:05b4542f) Enabled non-edge zones available for scan: 1 2025-06-29 03:05:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1ff6262c]) (logid:05b4542f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:05:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfdafcd4]) (logid:b38ec4c0) No inactive management server node found 2025-06-29 03:05:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfdafcd4]) (logid:b38ec4c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:05:57,909 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-512959cc]) (logid:b667a6ee) AutoScaling Monitor is running... 2025-06-29 03:05:57,920 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f50df45c]) (logid:cf75593e) HA health check task is running... 2025-06-29 03:05:58,030 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-49df21e4]) (logid:264a1215) Starting to remove all volume_stats rows older than [Sat Jun 28 15:05:58 CEST 2025]. 2025-06-29 03:05:58,031 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-49df21e4]) (logid:264a1215) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:05:58 CEST 2025]. 2025-06-29 03:05:59,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1a4436d]) (logid:b40dcbcd) No inactive management server node found 2025-06-29 03:05:59,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1a4436d]) (logid:b40dcbcd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:06:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd1f7860]) (logid:08f864cb) No inactive management server node found 2025-06-29 03:06:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd1f7860]) (logid:08f864cb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:01,921 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-41c9e862]) (logid:bd947bf4) HA health check task is running... 2025-06-29 03:06:02,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-076531a8]) (logid:31e1dae5) No inactive management server node found 2025-06-29 03:06:02,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-076531a8]) (logid:31e1dae5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:02,442 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:06:02,446 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:06:02,448 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) Seq 2-3048092522799696696: Routed from 90520738888109 2025-06-29 03:06:02,505 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) Seq 1-3048092522799696696: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:06:02,539 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:06:02,540 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) Seq 2-1612007191621929974: Routed from 90520738888109 2025-06-29 03:06:02,588 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f4047a32]) (logid:f77c9cdc) Seq 2-1612007191621929974: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:06:03,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4f7a839]) (logid:b1caafe0) No inactive management server node found 2025-06-29 03:06:03,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4f7a839]) (logid:b1caafe0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:05,119 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8018: Seq 7-8018: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:06:05,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8018 2025-06-29 03:06:05,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c681e9e6]) (logid:60fefea6) No inactive management server node found 2025-06-29 03:06:05,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c681e9e6]) (logid:60fefea6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:06:05,922 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c918d6ab]) (logid:c99bbace) HA health check task is running... 2025-06-29 03:06:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e0a727c]) (logid:e0cff87e) No inactive management server node found 2025-06-29 03:06:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e0a727c]) (logid:e0cff87e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:08,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80f0ec24]) (logid:b0a2af0a) No inactive management server node found 2025-06-29 03:06:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80f0ec24]) (logid:b0a2af0a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-886cf7aa]) (logid:35a797e6) No inactive management server node found 2025-06-29 03:06:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-886cf7aa]) (logid:35a797e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:09,924 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b57ab4ef]) (logid:b4c80846) HA health check task is running... 2025-06-29 03:06:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ded290fc]) (logid:5e88a285) Found 0 routers to update status. 2025-06-29 03:06:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ded290fc]) (logid:5e88a285) Found 0 VPC's to update Redundant State. 2025-06-29 03:06:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ded290fc]) (logid:5e88a285) Found 0 networks to update RvR status. 2025-06-29 03:06:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3babc47]) (logid:e316493b) No inactive management server node found 2025-06-29 03:06:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3babc47]) (logid:e316493b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cd28368]) (logid:306068b4) No inactive management server node found 2025-06-29 03:06:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cd28368]) (logid:306068b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:13,481 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-475a84df]) (logid:c514c58e) Starting to remove all vm_stats rows older than [Sat Jun 28 15:06:13 CEST 2025]. 2025-06-29 03:06:13,517 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-475a84df]) (logid:c514c58e) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:06:13 CEST 2025]. 2025-06-29 03:06:13,925 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d2a83429]) (logid:185102ec) HA health check task is running... 2025-06-29 03:06:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e173858]) (logid:f1dc6bff) No inactive management server node found 2025-06-29 03:06:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e173858]) (logid:f1dc6bff) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:06:15,119 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8019: Seq 7-8019: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:06:15,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8019 2025-06-29 03:06:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71c4c9a5]) (logid:a20f5e9f) No inactive management server node found 2025-06-29 03:06:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71c4c9a5]) (logid:a20f5e9f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a273a30]) (logid:4705c09e) No inactive management server node found 2025-06-29 03:06:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a273a30]) (logid:4705c09e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:17,926 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8c8e470d]) (logid:ed89522b) HA health check task is running... 2025-06-29 03:06:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05ccff36]) (logid:443e198b) No inactive management server node found 2025-06-29 03:06:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05ccff36]) (logid:443e198b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:20,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8020: Seq 7-8020: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:06:20,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8020 2025-06-29 03:06:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4c924af]) (logid:b6d1e448) No inactive management server node found 2025-06-29 03:06:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4c924af]) (logid:b6d1e448) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9e4591a]) (logid:ef866831) No inactive management server node found 2025-06-29 03:06:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9e4591a]) (logid:ef866831) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:21,927 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-12e02cac]) (logid:7f84e4a9) HA health check task is running... 2025-06-29 03:06:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bc2817a]) (logid:9bc2747b) No inactive management server node found 2025-06-29 03:06:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bc2817a]) (logid:9bc2747b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:06:24,209 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) StorageCollector is running... 2025-06-29 03:06:24,217 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:06:24,217 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:06:24,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:06:24,219 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-6717118844223095482: Routed from 90520738888109 2025-06-29 03:06:24,273 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 4-6717118844223095482: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:06:24,279 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:06:24,279 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:06:24,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:06:24,280 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-1612007191621929975: Routed from 90520738888109 2025-06-29 03:06:24,292 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-1612007191621929975: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:06:24,302 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:06:24,302 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:06:24,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:06:24,304 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-1612007191621929976: Routed from 90520738888109 2025-06-29 03:06:24,399 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-1612007191621929976: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:06:24,409 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:06:24,409 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:06:24,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:06:24,410 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-3048092522799696697: Routed from 90520738888109 2025-06-29 03:06:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b877aba]) (logid:7052c09c) No inactive management server node found 2025-06-29 03:06:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b877aba]) (logid:7052c09c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:25,460 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 1-3048092522799696697: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:06:25,462 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:06:25,462 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:06:25,462 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:06:25,463 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-1612007191621929977: Routed from 90520738888109 2025-06-29 03:06:25,928 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a00b4e40]) (logid:57c6d248) HA health check task is running... 2025-06-29 03:06:26,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ce4381e]) (logid:76c2c7ad) No inactive management server node found 2025-06-29 03:06:26,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ce4381e]) (logid:76c2c7ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:26,526 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-499ef73d]) (logid:f9bd5d0f) Seq 2-1612007191621929977: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:06:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5321d3e9]) (logid:5ba0f2b9) Enabled non-edge zones available for scan: 1 2025-06-29 03:06:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5321d3e9]) (logid:5ba0f2b9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:06:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5321d3e9]) (logid:5ba0f2b9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:06:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5ab3ebd6]) (logid:cd10eda9) Enabled non-edge zones available for scan: 1 2025-06-29 03:06:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5ab3ebd6]) (logid:cd10eda9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:06:27,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb28e180]) (logid:a27b870b) No inactive management server node found 2025-06-29 03:06:27,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb28e180]) (logid:a27b870b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5f509a7]) (logid:1a0ed0db) No inactive management server node found 2025-06-29 03:06:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5f509a7]) (logid:1a0ed0db) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:29,930 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4e4397dc]) (logid:26f93ded) HA health check task is running... 2025-06-29 03:06:30,099 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:06:30,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:06:30,100 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) Seq 2-3048092522799696698: Routed from 90520738888109 2025-06-29 03:06:30,102 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) Seq 1-3048092522799696698: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:06:30,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:06:30,104 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) Seq 2-1612007191621929978: Routed from 90520738888109 2025-06-29 03:06:30,106 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-1929c5f8]) (logid:fc35b659) Seq 2-1612007191621929978: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:06:30,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8021: Seq 7-8021: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:06:30,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8021 2025-06-29 03:06:30,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32486d91]) (logid:72ad4b68) No inactive management server node found 2025-06-29 03:06:30,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32486d91]) (logid:72ad4b68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:32,343 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3082d45b]) (logid:f6c7ea16) No inactive management server node found 2025-06-29 03:06:32,343 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3082d45b]) (logid:f6c7ea16) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283168dd]) (logid:2bdeef91) No inactive management server node found 2025-06-29 03:06:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283168dd]) (logid:2bdeef91) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:33,931 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2a60a849]) (logid:65b1f6ec) HA health check task is running... 2025-06-29 03:06:33,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:06:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:06:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:06:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:06:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-161fa403]) (logid:6fc802ec) No inactive management server node found 2025-06-29 03:06:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-161fa403]) (logid:6fc802ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-086f7d01]) (logid:1d3241ee) No inactive management server node found 2025-06-29 03:06:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-086f7d01]) (logid:1d3241ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:37,932 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-762bc775]) (logid:e2d8ad86) HA health check task is running... 2025-06-29 03:06:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d46e91f]) (logid:f48ec2b4) No inactive management server node found 2025-06-29 03:06:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d46e91f]) (logid:f48ec2b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a209ccc]) (logid:fa1087cb) No inactive management server node found 2025-06-29 03:06:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a209ccc]) (logid:fa1087cb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:40,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8022: Seq 7-8022: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:06:40,141 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8022 2025-06-29 03:06:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0d222355]) (logid:280456f4) Found 0 routers to update status. 2025-06-29 03:06:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0d222355]) (logid:280456f4) Found 0 VPC's to update Redundant State. 2025-06-29 03:06:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0d222355]) (logid:280456f4) Found 0 networks to update RvR status. 2025-06-29 03:06:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-937dd050]) (logid:edf52235) DbCollector is running... 2025-06-29 03:06:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fe26779]) (logid:13d765ba) No inactive management server node found 2025-06-29 03:06:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fe26779]) (logid:13d765ba) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:41,934 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8c154deb]) (logid:91a218c0) HA health check task is running... 2025-06-29 03:06:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) ManagementServerCollector is running... 2025-06-29 03:06:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) DB properties were already loaded 2025-06-29 03:06:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:06:42,168 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Execution of process [4062840] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:06:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Exit value of process [4062840] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:06:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Process [4062840] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:06:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) usage install: null 2025-06-29 03:06:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Sessions active: 0 2025-06-29 03:06:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:06:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062843] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:06:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:06:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062848] for command [/bin/bash -c uname -r ]. 2025-06-29 03:06:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:06:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062849] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:06:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) system memory from /proc: 4887515136 2025-06-29 03:06:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:06:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062854] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:06:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) free memory from /proc: 490381312 2025-06-29 03:06:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:06:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062859] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:06:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) used memory from /proc: 764508 2025-06-29 03:06:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:06:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062860] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:06:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:06:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062861] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:06:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:06:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062862] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:06:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:06:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062869] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:06:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:06:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Successfully executed process [4062870] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:06:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:06:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail buffersdirect.count, 120 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail buffersdirect.used, 657986 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail buffersmapped.count, 7 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail buffersmapped.used, 798602 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail gcPS-Scavenge.count, 500 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail gcPS-Scavenge.time, 3637 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail jvmuptime, 43142321 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memoryheap.committed, 354418688 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memoryheap.init, 77594624 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memoryheap.usage, 0.1027837269779615 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorynon-heap.max, -1 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorynon-heap.usage, -2.20594384E8 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorynon-heap.used, 220594384 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17611241169021594 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3029157779851368 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.Metaspace.usage, 0.9857866267012202 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.PS-Eden-Space.usage, 0.01962193444212941 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.PS-Old-Gen.usage, 0.12058866229174295 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorypools.PS-Survivor-Space.usage, 0.8315290368121603 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorytotal.committed, 579469312 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorytotal.init, 85262336 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorytotal.max, 1908932607 2025-06-29 03:06:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail memorytotal.used, 416863712 2025-06-29 03:06:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail threadsdeadlocks, [] 2025-06-29 03:06:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail threadsnew.count, 0 2025-06-29 03:06:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:06:42,248 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-1d0161d7]) (logid:3c944152) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:06:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":158962808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43142246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":416050264,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196217552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490381312,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368839,53809260,4587966875],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:06:42,248 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1437, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:06:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":158962808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43142246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":416050264,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196217552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490381312,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368839,53809260,4587966875],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:06:42,248 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:06:42,252 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-321:[ctx-1048403b]) (logid:69693c9b) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:06:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":158962808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43142246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":416050264,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196217552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490381312,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368839,53809260,4587966875],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:06:42,252 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:06:42,252 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1437, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:06:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354418688,"freeJvmMemoryBytes":158962808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43142246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":416050264,"totalCommitted":579469312,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196217552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490381312,"systemMemoryUsed":764508,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.08],"systemCyclesUsage":[66368839,53809260,4587966875],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:06:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37e68479]) (logid:b1516995) No inactive management server node found 2025-06-29 03:06:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37e68479]) (logid:b1516995) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c10c0d4]) (logid:774b9833) No inactive management server node found 2025-06-29 03:06:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c10c0d4]) (logid:774b9833) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:06:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:06:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:06:45,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2b868a5]) (logid:349e7d38) No inactive management server node found 2025-06-29 03:06:45,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2b868a5]) (logid:349e7d38) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:45,935 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-72ee83ce]) (logid:a584c713) HA health check task is running... 2025-06-29 03:06:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-678db0ca]) (logid:657e9d12) VM scheduler.poll is being called at 2025-06-29T01:07:00+0000 2025-06-29 03:06:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-678db0ca]) (logid:657e9d12) Got 0 scheduled jobs to be executed at 2025-06-29T01:07:00+0000 2025-06-29 03:06:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-678db0ca]) (logid:657e9d12) Cleaned up 0 VM scheduled job entries 2025-06-29 03:06:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-8bcc32c4]) (logid:a6a192b5) VM Operation Thread Running 2025-06-29 03:06:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3881b5a0]) (logid:4a56df68) ClusterDRS.poll is being called at 2025-06-29T01:07:00+0000 2025-06-29 03:06:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3881b5a0]) (logid:4a56df68) Removed 0 old drs migration plans 2025-06-29 03:06:47,206 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:06:47,208 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:06:47,209 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:06:47,212 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:06:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58eec0ef]) (logid:71d5ee63) No inactive management server node found 2025-06-29 03:06:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58eec0ef]) (logid:71d5ee63) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d4b6cef]) (logid:1572649a) No inactive management server node found 2025-06-29 03:06:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d4b6cef]) (logid:1572649a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:49,936 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7f81f633]) (logid:b622afce) HA health check task is running... 2025-06-29 03:06:50,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8024: Seq 7-8024: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:06:50,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8024 2025-06-29 03:06:50,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53830fc1]) (logid:0626ffb4) No inactive management server node found 2025-06-29 03:06:50,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53830fc1]) (logid:0626ffb4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfb0acde]) (logid:5658b22c) No inactive management server node found 2025-06-29 03:06:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfb0acde]) (logid:5658b22c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58bf24c7]) (logid:913f1caa) No inactive management server node found 2025-06-29 03:06:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58bf24c7]) (logid:913f1caa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:53,938 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4aa1aaec]) (logid:2d0bc881) HA health check task is running... 2025-06-29 03:06:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97c2b228]) (logid:02fdad20) No inactive management server node found 2025-06-29 03:06:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97c2b228]) (logid:02fdad20) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:56,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b13a635]) (logid:791aeded) No inactive management server node found 2025-06-29 03:06:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b13a635]) (logid:791aeded) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9d1eb3c5]) (logid:0b625831) Enabled non-edge zones available for scan: 1 2025-06-29 03:06:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9d1eb3c5]) (logid:0b625831) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:06:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9d1eb3c5]) (logid:0b625831) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:06:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fdd42306]) (logid:2d15d58c) Enabled non-edge zones available for scan: 1 2025-06-29 03:06:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fdd42306]) (logid:2d15d58c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:06:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c433b20]) (logid:597ddc4f) No inactive management server node found 2025-06-29 03:06:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c433b20]) (logid:597ddc4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:06:57,911 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-5a1c206f]) (logid:513a7730) AutoScaling Monitor is running... 2025-06-29 03:06:57,939 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-56d19c89]) (logid:183c1ede) HA health check task is running... 2025-06-29 03:06:58,032 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-a5beb80b]) (logid:3d152aba) Starting to remove all volume_stats rows older than [Sat Jun 28 15:06:58 CEST 2025]. 2025-06-29 03:06:58,033 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-a5beb80b]) (logid:3d152aba) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:06:58 CEST 2025]. 2025-06-29 03:06:59,340 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c774b71]) (logid:0fba0fcc) No inactive management server node found 2025-06-29 03:06:59,340 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c774b71]) (logid:0fba0fcc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:00,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8025: Seq 7-8025: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:00,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8025 2025-06-29 03:07:00,828 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4636f23a]) (logid:43d6f3d7) No inactive management server node found 2025-06-29 03:07:00,828 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4636f23a]) (logid:43d6f3d7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:01,940 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b30e0546]) (logid:07ecc5e3) HA health check task is running... 2025-06-29 03:07:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a70f5b82]) (logid:34d600a8) No inactive management server node found 2025-06-29 03:07:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a70f5b82]) (logid:34d600a8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:07:02,658 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:07:02,662 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:07:02,664 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) Seq 2-3048092522799696699: Routed from 90520738888109 2025-06-29 03:07:02,715 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) Seq 1-3048092522799696699: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:07:02,761 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:07:02,762 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) Seq 2-1612007191621929979: Routed from 90520738888109 2025-06-29 03:07:02,806 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-9170a752]) (logid:cd3fefca) Seq 2-1612007191621929979: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:07:03,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20416ec4]) (logid:d4bd2651) No inactive management server node found 2025-06-29 03:07:03,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20416ec4]) (logid:d4bd2651) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33459784]) (logid:7591e2a0) No inactive management server node found 2025-06-29 03:07:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33459784]) (logid:7591e2a0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:05,941 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-56edc465]) (logid:af341919) HA health check task is running... 2025-06-29 03:07:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7430c97d]) (logid:f455976a) No inactive management server node found 2025-06-29 03:07:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7430c97d]) (logid:f455976a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad66b463]) (logid:0a1e7f0d) No inactive management server node found 2025-06-29 03:07:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad66b463]) (logid:0a1e7f0d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:09,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1f1e25e]) (logid:6dd9f66d) No inactive management server node found 2025-06-29 03:07:09,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1f1e25e]) (logid:6dd9f66d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:09,943 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-013d99fb]) (logid:88785950) HA health check task is running... 2025-06-29 03:07:10,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8026: Seq 7-8026: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:10,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8026 2025-06-29 03:07:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f5179b4]) (logid:1cc36536) Found 0 routers to update status. 2025-06-29 03:07:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f5179b4]) (logid:1cc36536) Found 0 VPC's to update Redundant State. 2025-06-29 03:07:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f5179b4]) (logid:1cc36536) Found 0 networks to update RvR status. 2025-06-29 03:07:11,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-973d948f]) (logid:ed42be66) No inactive management server node found 2025-06-29 03:07:11,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-973d948f]) (logid:ed42be66) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccdf5da4]) (logid:f7c89410) No inactive management server node found 2025-06-29 03:07:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccdf5da4]) (logid:f7c89410) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:13,519 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-d8f9ee5e]) (logid:24158f19) Starting to remove all vm_stats rows older than [Sat Jun 28 15:07:13 CEST 2025]. 2025-06-29 03:07:13,556 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-d8f9ee5e]) (logid:24158f19) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:07:13 CEST 2025]. 2025-06-29 03:07:13,944 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b09b056d]) (logid:56ba258f) HA health check task is running... 2025-06-29 03:07:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96615822]) (logid:d26b7ce3) No inactive management server node found 2025-06-29 03:07:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96615822]) (logid:d26b7ce3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:15,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c402a25d]) (logid:1cff0302) No inactive management server node found 2025-06-29 03:07:15,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c402a25d]) (logid:1cff0302) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:17,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-6eb7d7ab]) (logid:549e20e9) Resetting hosts suitable for reconnect 2025-06-29 03:07:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-6eb7d7ab]) (logid:549e20e9) Completed resetting hosts suitable for reconnect 2025-06-29 03:07:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-6eb7d7ab]) (logid:549e20e9) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:07:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-6eb7d7ab]) (logid:549e20e9) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:07:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-6eb7d7ab]) (logid:549e20e9) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:07:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-6eb7d7ab]) (logid:549e20e9) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:07:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1989b928]) (logid:d2fc4a9d) No inactive management server node found 2025-06-29 03:07:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1989b928]) (logid:d2fc4a9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:17,945 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f41636f7]) (logid:eaee76ac) HA health check task is running... 2025-06-29 03:07:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02b24be2]) (logid:00073b99) No inactive management server node found 2025-06-29 03:07:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02b24be2]) (logid:00073b99) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:20,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8027: Seq 7-8027: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:20,143 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8027 2025-06-29 03:07:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4835d679]) (logid:6b5988e8) No inactive management server node found 2025-06-29 03:07:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4835d679]) (logid:6b5988e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:07:21,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4860297]) (logid:84d78eb5) No inactive management server node found 2025-06-29 03:07:21,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4860297]) (logid:84d78eb5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:21,947 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c3d2a3e9]) (logid:3ff8ec34) HA health check task is running... 2025-06-29 03:07:23,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d6674dc]) (logid:c80c8eaa) No inactive management server node found 2025-06-29 03:07:23,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d6674dc]) (logid:c80c8eaa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a981c79]) (logid:4a6de0bd) No inactive management server node found 2025-06-29 03:07:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a981c79]) (logid:4a6de0bd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:25,948 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-98bed23d]) (logid:0dfc596d) HA health check task is running... 2025-06-29 03:07:26,327 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a986777]) (logid:2d9b945d) No inactive management server node found 2025-06-29 03:07:26,328 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a986777]) (logid:2d9b945d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:26,527 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) StorageCollector is running... 2025-06-29 03:07:26,531 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:07:26,531 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:07:26,531 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:07:26,532 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-6717118844223095483: Routed from 90520738888109 2025-06-29 03:07:26,586 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 4-6717118844223095483: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:07:26,590 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:07:26,590 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:07:26,590 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:07:26,591 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-3048092522799696700: Routed from 90520738888109 2025-06-29 03:07:26,638 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 1-3048092522799696700: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:07:26,648 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:07:26,648 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:07:26,648 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:07:26,649 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-1612007191621929980: Routed from 90520738888109 2025-06-29 03:07:26,668 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-1612007191621929980: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:07:26,678 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:07:26,678 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:07:26,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:07:26,679 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-3048092522799696701: Routed from 90520738888109 2025-06-29 03:07:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bfa37604]) (logid:03ea9e88) Enabled non-edge zones available for scan: 1 2025-06-29 03:07:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bfa37604]) (logid:03ea9e88) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:07:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bfa37604]) (logid:03ea9e88) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:07:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f1318ff3]) (logid:a2447c31) Enabled non-edge zones available for scan: 1 2025-06-29 03:07:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f1318ff3]) (logid:a2447c31) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:07:27,730 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 1-3048092522799696701: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:07:27,732 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:07:27,732 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:07:27,732 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:07:27,732 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-1612007191621929981: Routed from 90520738888109 2025-06-29 03:07:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3db7d427]) (logid:d4a7275b) No inactive management server node found 2025-06-29 03:07:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3db7d427]) (logid:d4a7275b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:28,790 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-20ee9486]) (logid:a5af5e61) Seq 2-1612007191621929981: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:07:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ef91ad5]) (logid:d47388cd) No inactive management server node found 2025-06-29 03:07:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ef91ad5]) (logid:d47388cd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:07:29,950 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ce8d3493]) (logid:d207cef2) HA health check task is running... 2025-06-29 03:07:30,109 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:07:30,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:07:30,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) Seq 2-3048092522799696702: Routed from 90520738888109 2025-06-29 03:07:30,112 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) Seq 1-3048092522799696702: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:07:30,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:07:30,114 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) Seq 2-1612007191621929982: Routed from 90520738888109 2025-06-29 03:07:30,118 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e552379e]) (logid:a76884e6) Seq 2-1612007191621929982: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:07:30,123 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8028: Seq 7-8028: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:30,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8028 2025-06-29 03:07:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebeadb95]) (logid:8b0df0b2) No inactive management server node found 2025-06-29 03:07:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebeadb95]) (logid:8b0df0b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cabcc3e]) (logid:f4c8fed0) No inactive management server node found 2025-06-29 03:07:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cabcc3e]) (logid:f4c8fed0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-167b16e7]) (logid:e562e6bc) No inactive management server node found 2025-06-29 03:07:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-167b16e7]) (logid:e562e6bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:33,951 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-883713c3]) (logid:32ad6b04) HA health check task is running... 2025-06-29 03:07:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:07:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:07:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:07:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:07:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a86d06a2]) (logid:262a55fc) No inactive management server node found 2025-06-29 03:07:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a86d06a2]) (logid:262a55fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0539bb7b]) (logid:f40db8eb) No inactive management server node found 2025-06-29 03:07:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0539bb7b]) (logid:f40db8eb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:37,952 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9134e916]) (logid:a1050ad0) HA health check task is running... 2025-06-29 03:07:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b090dbbf]) (logid:0cd82509) No inactive management server node found 2025-06-29 03:07:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b090dbbf]) (logid:0cd82509) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:07:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a91336e2]) (logid:f6194fb2) No inactive management server node found 2025-06-29 03:07:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a91336e2]) (logid:f6194fb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:40,123 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8029: Seq 7-8029: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:40,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8029 2025-06-29 03:07:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9bab7907]) (logid:45f57240) Found 0 routers to update status. 2025-06-29 03:07:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9bab7907]) (logid:45f57240) Found 0 VPC's to update Redundant State. 2025-06-29 03:07:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9bab7907]) (logid:45f57240) Found 0 networks to update RvR status. 2025-06-29 03:07:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-afe7dd95]) (logid:04953047) DbCollector is running... 2025-06-29 03:07:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-710f298b]) (logid:c46e38ec) No inactive management server node found 2025-06-29 03:07:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-710f298b]) (logid:c46e38ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:07:41,953 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1387e7f7]) (logid:c19be55c) HA health check task is running... 2025-06-29 03:07:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) ManagementServerCollector is running... 2025-06-29 03:07:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) DB properties were already loaded 2025-06-29 03:07:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:07:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Execution of process [4062875] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:07:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Exit value of process [4062875] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:07:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Process [4062875] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:07:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) usage install: null 2025-06-29 03:07:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Sessions active: 0 2025-06-29 03:07:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:07:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062878] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:07:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:07:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062883] for command [/bin/bash -c uname -r ]. 2025-06-29 03:07:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:07:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062884] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:07:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) system memory from /proc: 4887515136 2025-06-29 03:07:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:07:42,201 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062889] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:07:42,201 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) free memory from /proc: 473378816 2025-06-29 03:07:42,201 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:07:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062894] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:07:42,215 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) used memory from /proc: 767648 2025-06-29 03:07:42,215 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:07:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062895] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:07:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:07:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062896] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:07:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:07:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062897] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:07:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:07:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062904] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:07:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:07:42,253 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Successfully executed process [4062905] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail buffersdirect.count, 120 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail buffersdirect.used, 657986 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail buffersmapped.count, 7 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail buffersmapped.used, 798602 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail gcPS-Scavenge.count, 501 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail gcPS-Scavenge.time, 3645 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail jvmuptime, 43202332 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memoryheap.committed, 354942976 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memoryheap.init, 77594624 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memoryheap.usage, 0.09968052680464244 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorynon-heap.max, -1 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorynon-heap.usage, -2.20594384E8 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorynon-heap.used, 220594384 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17611241169021594 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3029157779851368 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:07:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.Metaspace.usage, 0.9857866267012202 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.PS-Eden-Space.usage, 0.010694440012644852 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.PS-Old-Gen.usage, 0.12061726902920175 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorypools.PS-Survivor-Space.usage, 0.8138103485107422 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorytotal.committed, 579993600 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorytotal.init, 85262336 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorytotal.max, 1908932607 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail memorytotal.used, 410929776 2025-06-29 03:07:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail threadsdeadlocks, [] 2025-06-29 03:07:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail threadsnew.count, 0 2025-06-29 03:07:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:07:42,259 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-dea8e259]) (logid:ed2752e0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:07:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354942976,"freeJvmMemoryBytes":165411200,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43202246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":410126160,"totalCommitted":579993600,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190283408,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":473378816,"systemMemoryUsed":767648,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369168,53809560,4587989996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:07:42,259 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1439, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:07:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354942976,"freeJvmMemoryBytes":165411200,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43202246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":410126160,"totalCommitted":579993600,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190283408,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":473378816,"systemMemoryUsed":767648,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369168,53809560,4587989996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:07:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:07:42,264 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:07:42,264 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-322:[ctx-c0093e14]) (logid:2b2cd700) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:07:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354942976,"freeJvmMemoryBytes":165411200,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43202246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":410126160,"totalCommitted":579993600,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190283408,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":473378816,"systemMemoryUsed":767648,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369168,53809560,4587989996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:07:42,264 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1439, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:07:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":354942976,"freeJvmMemoryBytes":165411200,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43202246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":410126160,"totalCommitted":579993600,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":190283408,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":473378816,"systemMemoryUsed":767648,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369168,53809560,4587989996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:07:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd0e4ea9]) (logid:b8ec68c7) No inactive management server node found 2025-06-29 03:07:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd0e4ea9]) (logid:b8ec68c7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:44,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b45db12e]) (logid:ed3e3249) No inactive management server node found 2025-06-29 03:07:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b45db12e]) (logid:ed3e3249) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:45,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8030: Seq 7-8030: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:45,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8030 2025-06-29 03:07:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:07:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:07:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27f22670]) (logid:0fb50cd9) No inactive management server node found 2025-06-29 03:07:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27f22670]) (logid:0fb50cd9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:45,955 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-44540a5c]) (logid:017bdca4) HA health check task is running... 2025-06-29 03:07:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6bc18ee8]) (logid:3dab6ac3) VM scheduler.poll is being called at 2025-06-29T01:08:00+0000 2025-06-29 03:07:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6bc18ee8]) (logid:3dab6ac3) Got 0 scheduled jobs to be executed at 2025-06-29T01:08:00+0000 2025-06-29 03:07:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6bc18ee8]) (logid:3dab6ac3) Cleaned up 0 VM scheduled job entries 2025-06-29 03:07:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ac536f9c]) (logid:6cf67ac6) VM Operation Thread Running 2025-06-29 03:07:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1aceec45]) (logid:797bb527) ClusterDRS.poll is being called at 2025-06-29T01:08:00+0000 2025-06-29 03:07:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1aceec45]) (logid:797bb527) Removed 0 old drs migration plans 2025-06-29 03:07:47,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:07:47,241 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:07:47,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:07:47,245 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:07:47,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-551b36a5]) (logid:e7403c35) No inactive management server node found 2025-06-29 03:07:47,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-551b36a5]) (logid:e7403c35) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:07:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fd12ba8]) (logid:10eb351f) No inactive management server node found 2025-06-29 03:07:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fd12ba8]) (logid:10eb351f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:49,956 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8e055adc]) (logid:c7cc524f) HA health check task is running... 2025-06-29 03:07:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-853bddc2]) (logid:ae4e9292) No inactive management server node found 2025-06-29 03:07:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-853bddc2]) (logid:ae4e9292) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:51,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f591b399]) (logid:0e5693e5) No inactive management server node found 2025-06-29 03:07:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f591b399]) (logid:0e5693e5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:53,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ac97532]) (logid:597c6103) No inactive management server node found 2025-06-29 03:07:53,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ac97532]) (logid:597c6103) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:53,958 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2483c8a4]) (logid:29435f82) HA health check task is running... 2025-06-29 03:07:54,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71c56fdc]) (logid:991a92ad) No inactive management server node found 2025-06-29 03:07:54,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71c56fdc]) (logid:991a92ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:55,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8032: Seq 7-8032: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:07:55,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8032 2025-06-29 03:07:56,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebd6913f]) (logid:9426f604) No inactive management server node found 2025-06-29 03:07:56,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebd6913f]) (logid:9426f604) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d254f393]) (logid:7e0e4f2e) Enabled non-edge zones available for scan: 1 2025-06-29 03:07:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d254f393]) (logid:7e0e4f2e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:07:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d254f393]) (logid:7e0e4f2e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:07:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-16783f9b]) (logid:10e430af) Enabled non-edge zones available for scan: 1 2025-06-29 03:07:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-16783f9b]) (logid:10e430af) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:07:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-432d701f]) (logid:c8b7c989) No inactive management server node found 2025-06-29 03:07:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-432d701f]) (logid:c8b7c989) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:07:57,913 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-d3ec8fa4]) (logid:6d67bf35) AutoScaling Monitor is running... 2025-06-29 03:07:57,959 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a8a79d26]) (logid:73522a33) HA health check task is running... 2025-06-29 03:07:58,035 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-cf830cc2]) (logid:7b09e3c0) Starting to remove all volume_stats rows older than [Sat Jun 28 15:07:58 CEST 2025]. 2025-06-29 03:07:58,036 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-cf830cc2]) (logid:7b09e3c0) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:07:58 CEST 2025]. 2025-06-29 03:07:59,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4448fa59]) (logid:22cb6cd1) No inactive management server node found 2025-06-29 03:07:59,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4448fa59]) (logid:22cb6cd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d6f0809]) (logid:cd230bd1) No inactive management server node found 2025-06-29 03:08:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d6f0809]) (logid:cd230bd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:01,960 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-49449b10]) (logid:ee205182) HA health check task is running... 2025-06-29 03:08:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-515f24a4]) (logid:5fad49b8) No inactive management server node found 2025-06-29 03:08:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-515f24a4]) (logid:5fad49b8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:02,859 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:08:02,862 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:08:02,863 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) Seq 2-3048092522799696703: Routed from 90520738888109 2025-06-29 03:08:02,916 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) Seq 1-3048092522799696703: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:08:02,949 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:08:02,950 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) Seq 2-1612007191621929983: Routed from 90520738888109 2025-06-29 03:08:02,995 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-64b105d9]) (logid:e73d7979) Seq 2-1612007191621929983: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:08:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0e89a78]) (logid:f6d131e5) No inactive management server node found 2025-06-29 03:08:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0e89a78]) (logid:f6d131e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:05,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8033: Seq 7-8033: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:05,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8033 2025-06-29 03:08:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26c65f13]) (logid:74c081c5) No inactive management server node found 2025-06-29 03:08:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26c65f13]) (logid:74c081c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:05,962 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-99053ea7]) (logid:e62a964a) HA health check task is running... 2025-06-29 03:08:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0eddd25a]) (logid:d07a48bf) No inactive management server node found 2025-06-29 03:08:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0eddd25a]) (logid:d07a48bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:08,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbc9be1f]) (logid:f592ff2b) No inactive management server node found 2025-06-29 03:08:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbc9be1f]) (logid:f592ff2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16e19441]) (logid:106592a5) No inactive management server node found 2025-06-29 03:08:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16e19441]) (logid:106592a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:09,963 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c463be15]) (logid:d48c1358) HA health check task is running... 2025-06-29 03:08:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-485368c8]) (logid:fd1a22ec) Found 0 routers to update status. 2025-06-29 03:08:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-485368c8]) (logid:fd1a22ec) Found 0 VPC's to update Redundant State. 2025-06-29 03:08:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-485368c8]) (logid:fd1a22ec) Found 0 networks to update RvR status. 2025-06-29 03:08:11,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-911adcd1]) (logid:b3d1bb59) No inactive management server node found 2025-06-29 03:08:11,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-911adcd1]) (logid:b3d1bb59) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb0d564]) (logid:91cb7f48) No inactive management server node found 2025-06-29 03:08:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb0d564]) (logid:91cb7f48) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:13,558 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-13171a13]) (logid:e05e6ed8) Starting to remove all vm_stats rows older than [Sat Jun 28 15:08:13 CEST 2025]. 2025-06-29 03:08:13,593 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-13171a13]) (logid:e05e6ed8) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:08:13 CEST 2025]. 2025-06-29 03:08:13,964 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-951080b8]) (logid:c8c2faba) HA health check task is running... 2025-06-29 03:08:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a5fcebe]) (logid:d1ced25a) No inactive management server node found 2025-06-29 03:08:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a5fcebe]) (logid:d1ced25a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:15,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8034: Seq 7-8034: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:15,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8034 2025-06-29 03:08:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4de05381]) (logid:f06f23b2) No inactive management server node found 2025-06-29 03:08:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4de05381]) (logid:f06f23b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-387c5842]) (logid:71765702) No inactive management server node found 2025-06-29 03:08:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-387c5842]) (logid:71765702) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:17,965 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d9c19bdb]) (logid:7a680fc2) HA health check task is running... 2025-06-29 03:08:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c52002a]) (logid:e8e92087) No inactive management server node found 2025-06-29 03:08:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c52002a]) (logid:e8e92087) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:20,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a599caf3]) (logid:90184f17) No inactive management server node found 2025-06-29 03:08:20,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a599caf3]) (logid:90184f17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3826fa37]) (logid:30f0d36b) No inactive management server node found 2025-06-29 03:08:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3826fa37]) (logid:30f0d36b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:21,967 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5177553c]) (logid:75d931de) HA health check task is running... 2025-06-29 03:08:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c96877b4]) (logid:4b9df99a) No inactive management server node found 2025-06-29 03:08:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c96877b4]) (logid:4b9df99a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab30ca4c]) (logid:149b50dc) No inactive management server node found 2025-06-29 03:08:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab30ca4c]) (logid:149b50dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:25,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8035: Seq 7-8035: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:25,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8035 2025-06-29 03:08:25,968 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5d9d0ff5]) (logid:c15b8218) HA health check task is running... 2025-06-29 03:08:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b33c0ba]) (logid:9e2f0f88) No inactive management server node found 2025-06-29 03:08:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b33c0ba]) (logid:9e2f0f88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-13ae7b53]) (logid:71d72a26) Enabled non-edge zones available for scan: 1 2025-06-29 03:08:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-13ae7b53]) (logid:71d72a26) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:08:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-13ae7b53]) (logid:71d72a26) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:08:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f2ad8fc]) (logid:be7018ac) Enabled non-edge zones available for scan: 1 2025-06-29 03:08:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f2ad8fc]) (logid:be7018ac) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:08:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f0ece50]) (logid:2cc4865f) No inactive management server node found 2025-06-29 03:08:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f0ece50]) (logid:2cc4865f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:28,791 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) StorageCollector is running... 2025-06-29 03:08:28,796 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:08:28,796 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:08:28,796 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:08:28,797 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-6717118844223095484: Routed from 90520738888109 2025-06-29 03:08:28,850 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 4-6717118844223095484: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:08:28,854 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:08:28,854 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:08:28,854 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:08:28,855 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-1612007191621929984: Routed from 90520738888109 2025-06-29 03:08:28,864 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-1612007191621929984: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:08:28,874 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:08:28,874 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:08:28,874 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:08:28,875 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-1612007191621929985: Routed from 90520738888109 2025-06-29 03:08:28,968 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-1612007191621929985: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:08:28,979 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:08:28,979 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:08:28,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:08:28,980 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-3048092522799696704: Routed from 90520738888109 2025-06-29 03:08:29,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53dc190f]) (logid:0f820cc3) No inactive management server node found 2025-06-29 03:08:29,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53dc190f]) (logid:0f820cc3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:08:29,969 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-56abe0d5]) (logid:4f59ae66) HA health check task is running... 2025-06-29 03:08:30,032 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 1-3048092522799696704: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:08:30,033 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:08:30,033 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:08:30,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:08:30,034 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-1612007191621929986: Routed from 90520738888109 2025-06-29 03:08:30,120 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:08:30,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:08:30,122 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) Seq 2-3048092522799696705: Routed from 90520738888109 2025-06-29 03:08:30,124 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) Seq 1-3048092522799696705: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:08:30,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:08:30,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8036: Seq 7-8036: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:30,127 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) Seq 2-1612007191621929987: Routed from 90520738888109 2025-06-29 03:08:30,130 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-12fd50a9]) (logid:685d7e7e) Seq 2-1612007191621929987: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:08:30,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8036 2025-06-29 03:08:30,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59156639]) (logid:d4f8eb21) No inactive management server node found 2025-06-29 03:08:30,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59156639]) (logid:d4f8eb21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:31,096 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90691b27]) (logid:a7297073) Seq 2-1612007191621929986: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:08:32,357 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae59a564]) (logid:1b309e1d) No inactive management server node found 2025-06-29 03:08:32,357 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae59a564]) (logid:1b309e1d) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:33,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b447f78]) (logid:67141b29) No inactive management server node found 2025-06-29 03:08:33,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b447f78]) (logid:67141b29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:33,970 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-437fcde7]) (logid:c2f37a1c) HA health check task is running... 2025-06-29 03:08:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:08:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:08:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:08:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:08:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36778138]) (logid:e9ba3525) No inactive management server node found 2025-06-29 03:08:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36778138]) (logid:e9ba3525) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a39e076c]) (logid:fd544d06) No inactive management server node found 2025-06-29 03:08:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a39e076c]) (logid:fd544d06) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:37,972 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5452341f]) (logid:d4e54acb) HA health check task is running... 2025-06-29 03:08:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6bf0e22]) (logid:a1f332a9) No inactive management server node found 2025-06-29 03:08:38,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6bf0e22]) (logid:a1f332a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-105543bf]) (logid:ff1e1fca) No inactive management server node found 2025-06-29 03:08:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-105543bf]) (logid:ff1e1fca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:40,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8037: Seq 7-8037: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:40,147 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8037 2025-06-29 03:08:40,816 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-666ca9ce]) (logid:deb06c49) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:08:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-666ca9ce]) (logid:deb06c49) Snapshot scheduler is being called at [2025-06-29T01:08:40+0000]. 2025-06-29 03:08:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-666ca9ce]) (logid:deb06c49) There are [0] scheduled snapshots to be executed at [2025-06-29T01:08:40+0000]. 2025-06-29 03:08:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-24c950d3]) (logid:7653a101) External devices are not used. Skipping external device usage collection 2025-06-29 03:08:40,934 DEBUG [c.c.a.m.A.AgentNewConnectionsMonitorTask] (NewAgentConnectionsMonitor-1:[ctx-f4117f14]) (logid:759102ff) Currently 0 active new connections, of which 0 have expired - [] 2025-06-29 03:08:40,946 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-93930f35]) (logid:bac4e4df) checking health of usage server 2025-06-29 03:08:40,948 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-93930f35]) (logid:bac4e4df) usage server running? false, heartbeat: null 2025-06-29 03:08:40,959 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-93930f35]) (logid:bac4e4df) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 03:08:40,960 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-93930f35]) (logid:bac4e4df) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 03:08:40,978 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-412f90c5]) (logid:2af78786) Backup backup.poll is being called at 2025-06-29T01:08:40+0000 2025-06-29 03:08:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-b57563fb]) (logid:70d3e6a5) Found 0 running routers. 2025-06-29 03:08:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e01b722f]) (logid:a5fe1ead) Found 0 routers to update status. 2025-06-29 03:08:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e01b722f]) (logid:a5fe1ead) Found 0 VPC's to update Redundant State. 2025-06-29 03:08:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e01b722f]) (logid:a5fe1ead) Found 0 networks to update RvR status. 2025-06-29 03:08:41,015 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-9068c89c]) (logid:070ad359) Found 1 running routers. 2025-06-29 03:08:41,024 INFO [c.c.n.v.V.VpcCleanupTask] (VpcChecker-1:[ctx-bd1cb743]) (logid:46cec46e) Found 0 removed VPCs to cleanup 2025-06-29 03:08:41,027 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-9068c89c]) (logid:070ad359) Wait time setting on com.cloud.agent.api.routing.GetRouterAlertsCommand is 1800 seconds 2025-06-29 03:08:41,029 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-9068c89c]) (logid:070ad359) Seq 2-1612007191621929988: Routed from 90520738888109 2025-06-29 03:08:41,030 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-9068c89c]) (logid:070ad359) Seq 2-1612007191621929988: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterAlertsCommand":{"previousAlertTimeStamp":"2025-06-20 12:09:07","accessDetails":{"router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:41,047 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-7c4c4fa6]) (logid:47c5aceb) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 03:08:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-062332a4]) (logid:323dd127) DbCollector is running... 2025-06-29 03:08:41,282 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-2e3439c2]) (logid:d235d851) Started resource counters recalculation periodic task. 2025-06-29 03:08:41,286 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-2e3439c2]) (logid:d235d851) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:08:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db0757d4]) (logid:0505d6d5) No inactive management server node found 2025-06-29 03:08:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db0757d4]) (logid:0505d6d5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:08:41,568 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 2-1612007191621929988: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetRouterAlertsAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:41,568 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-9068c89c]) (logid:070ad359) Seq 2-1612007191621929988: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterAlertsAnswer } } 2025-06-29 03:08:41,570 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Found 1 running routers. 2025-06-29 03:08:41,574 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:08:41,575 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Seq 2-1612007191621929989: Routed from 90520738888109 2025-06-29 03:08:41,575 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Seq 2-1612007191621929989: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:41,643 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-2e3439c2]) (logid:d235d851) Finished resource counters recalculation periodic task. 2025-06-29 03:08:41,841 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[]) (logid:) Seq 2-1612007191621929989: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:41,841 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Seq 2-1612007191621929989: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:08:41,841 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 03:08:41,844 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 03:08:41,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 03:08:41,884 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Seq 2-1612007191621929990: Routed from 90520738888109 2025-06-29 03:08:41,885 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Seq 2-1612007191621929990: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:41,973 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1e1ff550]) (logid:de41f936) HA health check task is running... 2025-06-29 03:08:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) ManagementServerCollector is running... 2025-06-29 03:08:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) DB properties were already loaded 2025-06-29 03:08:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:08:42,170 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Execution of process [4062919] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:08:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Exit value of process [4062919] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:08:42,170 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Process [4062919] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:08:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) usage install: null 2025-06-29 03:08:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Sessions active: 0 2025-06-29 03:08:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:08:42,178 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062922] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:08:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:08:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062927] for command [/bin/bash -c uname -r ]. 2025-06-29 03:08:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:08:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062928] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:08:42,190 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) system memory from /proc: 4887515136 2025-06-29 03:08:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:08:42,197 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062933] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:08:42,197 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) free memory from /proc: 484401152 2025-06-29 03:08:42,197 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:08:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062938] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:08:42,210 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) used memory from /proc: 767776 2025-06-29 03:08:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:08:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062939] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:08:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:08:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062940] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:08:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:08:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062941] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:08:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:08:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062948] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:08:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:08:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Successfully executed process [4062949] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:08:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail buffersdirect.capacity, 657986 2025-06-29 03:08:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail buffersdirect.count, 120 2025-06-29 03:08:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail buffersdirect.used, 657986 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail buffersmapped.count, 7 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail buffersmapped.used, 798602 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail gcPS-Scavenge.count, 502 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail gcPS-Scavenge.time, 3654 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail jvmuptime, 43262328 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memoryheap.committed, 355467264 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memoryheap.init, 77594624 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memoryheap.usage, 0.10486533425071022 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorynon-heap.max, -1 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorynon-heap.usage, -2.20619856E8 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorynon-heap.used, 220619856 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1762540405891762 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.Metaspace.usage, 0.9857866267012202 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.PS-Eden-Space.usage, 0.024878015619797612 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.PS-Old-Gen.usage, 0.12065159711415233 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorypools.PS-Survivor-Space.usage, 0.78750244140625 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorytotal.committed, 580517888 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorytotal.init, 85262336 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorytotal.max, 1908932607 2025-06-29 03:08:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail memorytotal.used, 420855000 2025-06-29 03:08:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail threadsdeadlocks, [] 2025-06-29 03:08:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail threadsnew.count, 0 2025-06-29 03:08:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:08:42,255 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-7f73b7a5]) (logid:45a95b32) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:08:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":156053216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43262247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":420033904,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200180856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484401152,"systemMemoryUsed":767776,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369516,53809881,4588013063],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:08:42,256 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1441, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:08:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":156053216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43262247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":420033904,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200180856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484401152,"systemMemoryUsed":767776,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369516,53809881,4588013063],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:08:42,256 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:08:42,259 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-322:[ctx-59a41bc2]) (logid:ff18da82) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:08:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":156053216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43262247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":420033904,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200180856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484401152,"systemMemoryUsed":767776,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369516,53809881,4588013063],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:08:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:08:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1441, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:08:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":156053216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43262247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":420033904,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":200180856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484401152,"systemMemoryUsed":767776,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.07],"systemCyclesUsage":[66369516,53809881,4588013063],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:08:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-295c1332]) (logid:b80a954c) No inactive management server node found 2025-06-29 03:08:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-295c1332]) (logid:b80a954c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:43,373 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 2-1612007191621929990: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.d2bf9623-b5c9-4126-ac9a-5034492115b7","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:43,373 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-74e459c5]) (logid:d059e490) Seq 2-1612007191621929990: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 03:08:43,375 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 03:08:43,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:08:43,379 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Seq 2-1612007191621929991: Routed from 90520738888109 2025-06-29 03:08:43,379 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Seq 2-1612007191621929991: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:43,815 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 2-1612007191621929991: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "13.730049133300781", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "26.022672653198242", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "37.78076171875", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "50.604820251464844", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159162217", "lastRunDuration": "45.27163505554199", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159162288", "lastRunDuration": "8094.242334365845", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159170426", "lastRunDuration": "68.75205039978027", "message": "Memory Usage within limits with current at 50.8783%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159170495", "lastRunDuration": "3234.586477279663", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159173768", "lastRunDuration": "39.59083557128906", "message": "Sufficient free space is 2571.5390625 MB"}, "lastRun": {"start": "2025-06-29 01:06:02.025769", "end": "2025-06-29 01:06:13.807928", "duration": "11.782159566879272"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158802124", "lastRunDuration": "83.99271965026855", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158802209", "lastRunDuration": "111.38057708740234", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158802578", "lastRunDuration": "78.51457595825195", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158802756", "lastRunDuration": "36.66257858276367", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:00:02.122386", "end": "2025-06-29 01:00:02.829943", "duration": "0.7075574398040771"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "13.730049133300781", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "26.022672653198242", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "37.78076171875", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "50.604820251464844", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159162217", "lastRunDuration": "45.27163505554199", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159162288", "lastRunDuration": "8094.242334365845", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159170426", "lastRunDuration": "68.75205039978027", "message": "Memory Usage within limits with current at 50.8783%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159170495", "lastRunDuration": "3234.586477279663", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159173768", "lastRunDuration": "39.59083557128906", "message": "Sufficient free space is 2571.5390625 MB"}, "lastRun": {"start": "2025-06-29 01:06:02.025769", "end": "2025-06-29 01:06:13.807928", "duration": "11.782159566879272"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158802124", "lastRunDuration": "83.99271965026855", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158802209", "lastRunDuration": "111.38057708740234", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158802578", "lastRunDuration": "78.51457595825195", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158802756", "lastRunDuration": "36.66257858276367", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:00:02.122386", "end": "2025-06-29 01:00:02.829943", "duration": "0.7075574398040771"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:43,816 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Seq 2-1612007191621929991: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:08:43,816 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "13.730049133300781", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "26.022672653198242", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "37.78076171875", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "50.604820251464844", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159162217", "lastRunDuration": "45.27163505554199", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159162288", "lastRunDuration": "8094.242334365845", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159170426", "lastRunDuration": "68.75205039978027", "message": "Memory Usage within limits with current at 50.8783%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159170495", "lastRunDuration": "3234.586477279663", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159173768", "lastRunDuration": "39.59083557128906", "message": "Sufficient free space is 2571.5390625 MB"}, "lastRun": {"start": "2025-06-29 01:06:02.025769", "end": "2025-06-29 01:06:13.807928", "duration": "11.782159566879272"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158802124", "lastRunDuration": "83.99271965026855", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158802209", "lastRunDuration": "111.38057708740234", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158802578", "lastRunDuration": "78.51457595825195", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158802756", "lastRunDuration": "36.66257858276367", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:00:02.122386", "end": "2025-06-29 01:00:02.829943", "duration": "0.7075574398040771"}}} 2025-06-29 03:08:43,860 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "13.730049133300781", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "26.022672653198242", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "37.78076171875", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159162028", "lastRunDuration": "50.604820251464844", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159162217", "lastRunDuration": "45.27163505554199", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159162288", "lastRunDuration": "8094.242334365845", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159170426", "lastRunDuration": "68.75205039978027", "message": "Memory Usage within limits with current at 50.8783%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159170495", "lastRunDuration": "3234.586477279663", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159173768", "lastRunDuration": "39.59083557128906", "message": "Sufficient free space is 2571.5390625 MB"}, "lastRun": {"start": "2025-06-29 01:06:02.025769", "end": "2025-06-29 01:06:13.807928", "duration": "11.782159566879272"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751158802124", "lastRunDuration": "83.99271965026855", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751158802209", "lastRunDuration": "111.38057708740234", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751158802578", "lastRunDuration": "78.51457595825195", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751158802756", "lastRunDuration": "36.66257858276367", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:00:02.122386", "end": "2025-06-29 01:00:02.829943", "duration": "0.7075574398040771"}}} 2025-06-29 03:08:43,862 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found check types executed on VR basic, start: 2025-06-29 01:06:02.025769, end: 2025-06-29 01:06:13.807928, duration: 11.782159566879272 2025-06-29 03:08:43,871 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1ae8f4ac- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 03:06:02 CEST 2025, details: service is running which took running duration (ms) 13.730049133300781 2025-06-29 03:08:43,883 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@a171648- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 03:06:02 CEST 2025, details: service is running which took running duration (ms) 26.022672653198242 2025-06-29 03:08:43,892 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6337f9a2- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 03:06:02 CEST 2025, details: service is running which took running duration (ms) 37.78076171875 2025-06-29 03:08:43,902 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@69bed5a1- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 03:06:02 CEST 2025, details: service is running which took running duration (ms) 50.604820251464844 2025-06-29 03:08:43,914 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@34ae01ae- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 03:06:02 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 45.27163505554199 2025-06-29 03:08:43,926 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6b9f425- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 03:06:02 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8094.242334365845 2025-06-29 03:08:43,935 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5a1ff7b9- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 03:06:10 CEST 2025, details: Memory Usage within limits with current at 50.8783% which took running duration (ms) 68.75205039978027 2025-06-29 03:08:43,946 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@45932286- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 03:06:10 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3234.586477279663 2025-06-29 03:08:43,955 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@174f8eb- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 03:06:13 CEST 2025, details: Sufficient free space is 2571.5390625 MB which took running duration (ms) 39.59083557128906 2025-06-29 03:08:43,955 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found check types executed on VR advanced, start: 2025-06-29 01:00:02.122386, end: 2025-06-29 01:00:02.829943, duration: 0.7075574398040771 2025-06-29 03:08:43,967 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@48253c3c- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 03:00:02 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 83.99271965026855 2025-06-29 03:08:43,977 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6da2a8be- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 03:00:02 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 111.38057708740234 2025-06-29 03:08:43,987 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@31f9309e- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 03:00:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 78.51457595825195 2025-06-29 03:08:43,998 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-04be3b2b]) (logid:c126e51e) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@17ec35ea- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 03:00:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 36.66257858276367 2025-06-29 03:08:44,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8db7f64]) (logid:7184cfe7) No inactive management server node found 2025-06-29 03:08:44,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8db7f64]) (logid:7184cfe7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:08:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:08:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6a77b27]) (logid:add609ba) No inactive management server node found 2025-06-29 03:08:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6a77b27]) (logid:add609ba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:45,974 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-69a2688a]) (logid:3463250d) HA health check task is running... 2025-06-29 03:08:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-12226b6a]) (logid:19da6bdb) VM scheduler.poll is being called at 2025-06-29T01:09:00+0000 2025-06-29 03:08:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-12226b6a]) (logid:19da6bdb) Got 0 scheduled jobs to be executed at 2025-06-29T01:09:00+0000 2025-06-29 03:08:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-12226b6a]) (logid:19da6bdb) Cleaned up 0 VM scheduled job entries 2025-06-29 03:08:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-6c9f2ef2]) (logid:312d5e39) VM Operation Thread Running 2025-06-29 03:08:46,185 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-1:[ctx-b6450b1b]) (logid:3636ea6b) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:08:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a30af60c]) (logid:5e8e32fa) ClusterDRS.poll is being called at 2025-06-29T01:09:00+0000 2025-06-29 03:08:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a30af60c]) (logid:5e8e32fa) Removed 0 old drs migration plans 2025-06-29 03:08:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1ee3c602]) (logid:bd9a199d) Resetting hosts suitable for reconnect 2025-06-29 03:08:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1ee3c602]) (logid:bd9a199d) Completed resetting hosts suitable for reconnect 2025-06-29 03:08:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1ee3c602]) (logid:bd9a199d) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:08:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1ee3c602]) (logid:bd9a199d) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:08:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1ee3c602]) (logid:bd9a199d) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:08:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1ee3c602]) (logid:bd9a199d) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:08:47,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:08:47,212 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:08:47,214 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:08:47,216 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:08:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14aab423]) (logid:36b930e1) No inactive management server node found 2025-06-29 03:08:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14aab423]) (logid:36b930e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:48,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27a88d75]) (logid:f12beb8d) No inactive management server node found 2025-06-29 03:08:48,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27a88d75]) (logid:f12beb8d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:49,975 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-45bdd89a]) (logid:5ef4a15c) HA health check task is running... 2025-06-29 03:08:50,127 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8039: Seq 7-8039: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:50,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8039 2025-06-29 03:08:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd15e097]) (logid:c366decf) No inactive management server node found 2025-06-29 03:08:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd15e097]) (logid:c366decf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:51,032 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-42d99637]) (logid:7dce5202) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:08:51,048 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-42d99637]) (logid:7dce5202) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:08:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b96ae8b]) (logid:d3c07f6f) No inactive management server node found 2025-06-29 03:08:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b96ae8b]) (logid:d3c07f6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7faeb506]) (logid:0eb72b91) No inactive management server node found 2025-06-29 03:08:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7faeb506]) (logid:0eb72b91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:53,977 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-95a7683a]) (logid:aa9ab6e0) HA health check task is running... 2025-06-29 03:08:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f246fd6]) (logid:e4d26d4a) No inactive management server node found 2025-06-29 03:08:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f246fd6]) (logid:e4d26d4a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:55,127 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8040: Seq 7-8040: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:08:55,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8040 2025-06-29 03:08:56,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:08:56,224 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Seq 2-3048092522799696706: Routed from 90520738888109 2025-06-29 03:08:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cf1667d]) (logid:69b91e22) No inactive management server node found 2025-06-29 03:08:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cf1667d]) (logid:69b91e22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c8a7b677]) (logid:0c15181d) Enabled non-edge zones available for scan: 1 2025-06-29 03:08:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c8a7b677]) (logid:0c15181d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:08:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c8a7b677]) (logid:0c15181d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:08:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-da9910f5]) (logid:f4747f5a) Enabled non-edge zones available for scan: 1 2025-06-29 03:08:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-da9910f5]) (logid:f4747f5a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:08:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e90decb]) (logid:00f60003) No inactive management server node found 2025-06-29 03:08:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e90decb]) (logid:00f60003) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:08:57,915 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-9f65379b]) (logid:19c725d3) AutoScaling Monitor is running... 2025-06-29 03:08:57,978 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d932fecf]) (logid:ffb919d0) HA health check task is running... 2025-06-29 03:08:58,038 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-e4ebf777]) (logid:11246bae) Starting to remove all volume_stats rows older than [Sat Jun 28 15:08:58 CEST 2025]. 2025-06-29 03:08:58,038 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-e4ebf777]) (logid:11246bae) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:08:58 CEST 2025]. 2025-06-29 03:08:59,290 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Seq 1-3048092522799696706: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:08:59,290 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 03:08:59,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:08:59,324 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Seq 2-1612007191621929992: Routed from 90520738888109 2025-06-29 03:08:59,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b13dbac7]) (logid:ed5fa6c6) No inactive management server node found 2025-06-29 03:08:59,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b13dbac7]) (logid:ed5fa6c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ac3a563]) (logid:10a02393) No inactive management server node found 2025-06-29 03:09:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ac3a563]) (logid:10a02393) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:01,979 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d59a5cca]) (logid:887c6469) HA health check task is running... 2025-06-29 03:09:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e186811]) (logid:24ec6f48) No inactive management server node found 2025-06-29 03:09:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e186811]) (logid:24ec6f48) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:03,057 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:09:03,060 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:09:03,060 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) Seq 2-3048092522799696707: Routed from 90520738888109 2025-06-29 03:09:03,077 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) Seq 1-3048092522799696707: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:09:03,111 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:09:03,111 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) Seq 2-1612007191621929993: Routed from 90520738888109 2025-06-29 03:09:03,163 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6c76b3b1]) (logid:1527b168) Seq 2-1612007191621929993: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:09:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc8a23a1]) (logid:cf873bb0) No inactive management server node found 2025-06-29 03:09:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc8a23a1]) (logid:cf873bb0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:04,438 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Seq 2-1612007191621929992: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:09:04,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-360e9d15]) (logid:6a533970) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 03:09:05,127 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8041: Seq 7-8041: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:09:05,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8041 2025-06-29 03:09:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b172748]) (logid:dfb12a2e) No inactive management server node found 2025-06-29 03:09:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b172748]) (logid:dfb12a2e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:09:05,980 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-edf9e4b6]) (logid:3ba59573) HA health check task is running... 2025-06-29 03:09:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfeef603]) (logid:694211a8) No inactive management server node found 2025-06-29 03:09:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfeef603]) (logid:694211a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:08,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd3705dd]) (logid:2a3557bc) No inactive management server node found 2025-06-29 03:09:08,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd3705dd]) (logid:2a3557bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74385057]) (logid:15b2ed05) No inactive management server node found 2025-06-29 03:09:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74385057]) (logid:15b2ed05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:09,982 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-06673196]) (logid:5489eaad) HA health check task is running... 2025-06-29 03:09:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f9d90f46]) (logid:3fe2b46d) Found 0 routers to update status. 2025-06-29 03:09:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f9d90f46]) (logid:3fe2b46d) Found 0 VPC's to update Redundant State. 2025-06-29 03:09:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f9d90f46]) (logid:3fe2b46d) Found 0 networks to update RvR status. 2025-06-29 03:09:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Running Capacity Checker ... 2025-06-29 03:09:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) recalculating system capacity 2025-06-29 03:09:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Executing cpu/ram capacity update 2025-06-29 03:09:11,071 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:11,078 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:09:11,081 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:09:11,082 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:09:11,093 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:11,093 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:11,097 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:09:11,098 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:09:11,099 DEBUG [c.c.c.CapacityManagerImpl] (pool-316-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:09:11,107 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Done executing cpu/ram capacity update 2025-06-29 03:09:11,107 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Executing storage capacity update 2025-06-29 03:09:11,112 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:09:11,113 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:09:11,113 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:09:11,115 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:09:11,121 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:09:11,121 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:09:11,121 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:09:11,123 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:09:11,128 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:09:11,128 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:09:11,128 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:09:11,131 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:09:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:09:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:09:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:09:11,138 DEBUG [c.c.s.StorageManagerImpl] (pool-317-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:09:11,139 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Done executing storage capacity update 2025-06-29 03:09:11,139 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Executing capacity updates for public ip and Vlans 2025-06-29 03:09:11,154 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Done capacity updates for public ip and Vlans 2025-06-29 03:09:11,154 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Executing capacity updates for private ip 2025-06-29 03:09:11,160 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Done executing capacity updates for private ip 2025-06-29 03:09:11,160 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Done recalculating system capacity 2025-06-29 03:09:11,189 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7d32f386]) (logid:ff108594) Done running Capacity Checker ... 2025-06-29 03:09:11,259 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-bbaed22d]) (logid:34eedae2) Checking if any host reservation can be released ... 2025-06-29 03:09:11,263 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-bbaed22d]) (logid:34eedae2) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:11,266 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-bbaed22d]) (logid:34eedae2) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:11,266 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-bbaed22d]) (logid:34eedae2) Done running HostReservationReleaseChecker ... 2025-06-29 03:09:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a62190ef]) (logid:c6692d01) No inactive management server node found 2025-06-29 03:09:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a62190ef]) (logid:c6692d01) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb9f7718]) (logid:27ee7350) No inactive management server node found 2025-06-29 03:09:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb9f7718]) (logid:27ee7350) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:13,595 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-d63a2e2e]) (logid:87b9a089) Starting to remove all vm_stats rows older than [Sat Jun 28 15:09:13 CEST 2025]. 2025-06-29 03:09:13,630 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-d63a2e2e]) (logid:87b9a089) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:09:13 CEST 2025]. 2025-06-29 03:09:13,983 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d3b4f2f9]) (logid:481838f5) HA health check task is running... 2025-06-29 03:09:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45738809]) (logid:cf229038) No inactive management server node found 2025-06-29 03:09:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45738809]) (logid:cf229038) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:15,128 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8042: Seq 7-8042: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:09:15,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8042 2025-06-29 03:09:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9651efe3]) (logid:febbedf7) No inactive management server node found 2025-06-29 03:09:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9651efe3]) (logid:febbedf7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:17,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aebf4415]) (logid:096c852d) No inactive management server node found 2025-06-29 03:09:17,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aebf4415]) (logid:096c852d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:09:17,984 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-106891bc]) (logid:553e4e9b) HA health check task is running... 2025-06-29 03:09:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-336eafbd]) (logid:a181de09) No inactive management server node found 2025-06-29 03:09:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-336eafbd]) (logid:a181de09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0d74a64]) (logid:150329ce) No inactive management server node found 2025-06-29 03:09:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0d74a64]) (logid:150329ce) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:21,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ba9f288]) (logid:36af33f4) No inactive management server node found 2025-06-29 03:09:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ba9f288]) (logid:36af33f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:21,985 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7814647f]) (logid:6b20366a) HA health check task is running... 2025-06-29 03:09:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17e50c65]) (logid:ab7fe37b) No inactive management server node found 2025-06-29 03:09:23,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17e50c65]) (logid:ab7fe37b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:09:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ae7082c]) (logid:91d97506) No inactive management server node found 2025-06-29 03:09:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ae7082c]) (logid:91d97506) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:25,128 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8043: Seq 7-8043: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:09:25,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8043 2025-06-29 03:09:25,987 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-62a634da]) (logid:9ecdccea) HA health check task is running... 2025-06-29 03:09:26,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c810716]) (logid:27a36f42) No inactive management server node found 2025-06-29 03:09:26,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c810716]) (logid:27a36f42) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-19e2b03a]) (logid:84d05fe8) Enabled non-edge zones available for scan: 1 2025-06-29 03:09:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-19e2b03a]) (logid:84d05fe8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:09:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-19e2b03a]) (logid:84d05fe8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:09:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-74390186]) (logid:cabc9c17) Enabled non-edge zones available for scan: 1 2025-06-29 03:09:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-74390186]) (logid:cabc9c17) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:09:27,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2355b86e]) (logid:90ea219d) No inactive management server node found 2025-06-29 03:09:27,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2355b86e]) (logid:90ea219d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:29,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3e66c5d]) (logid:a117bb08) No inactive management server node found 2025-06-29 03:09:29,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3e66c5d]) (logid:a117bb08) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:09:29,988 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-be984262]) (logid:f5b4e22b) HA health check task is running... 2025-06-29 03:09:30,132 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:09:30,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:09:30,135 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) Seq 2-3048092522799696708: Routed from 90520738888109 2025-06-29 03:09:30,177 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) Seq 1-3048092522799696708: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:09:30,178 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:09:30,179 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) Seq 2-1612007191621929994: Routed from 90520738888109 2025-06-29 03:09:30,182 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-31b06fd8]) (logid:452cfbda) Seq 2-1612007191621929994: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:09:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73c55ca9]) (logid:7f1b5c28) No inactive management server node found 2025-06-29 03:09:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73c55ca9]) (logid:7f1b5c28) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:31,097 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) StorageCollector is running... 2025-06-29 03:09:31,102 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:09:31,102 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:09:31,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:09:31,103 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-6717118844223095485: Routed from 90520738888109 2025-06-29 03:09:31,165 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 4-6717118844223095485: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:09:31,169 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:09:31,169 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:09:31,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:09:31,170 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-1612007191621929995: Routed from 90520738888109 2025-06-29 03:09:31,221 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-1612007191621929995: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:09:31,244 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:09:31,244 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:09:31,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:09:31,245 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-1612007191621929996: Routed from 90520738888109 2025-06-29 03:09:31,336 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-1612007191621929996: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:09:31,347 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:09:31,347 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:09:31,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:09:31,348 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-3048092522799696709: Routed from 90520738888109 2025-06-29 03:09:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6825a964]) (logid:979f5764) No inactive management server node found 2025-06-29 03:09:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6825a964]) (logid:979f5764) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:09:32,397 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 1-3048092522799696709: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:09:32,399 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:09:32,399 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:09:32,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:09:32,400 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-1612007191621929997: Routed from 90520738888109 2025-06-29 03:09:33,467 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-57d4372e]) (logid:a4680a75) Seq 2-1612007191621929997: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:09:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b529524c]) (logid:b23ea2d8) No inactive management server node found 2025-06-29 03:09:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b529524c]) (logid:b23ea2d8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:33,989 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:33,989 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-446cd66b]) (logid:ecd44002) HA health check task is running... 2025-06-29 03:09:33,991 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:33,992 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:09:33,994 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:09:35,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8044: Seq 7-8044: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:09:35,150 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8044 2025-06-29 03:09:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1410bdae]) (logid:a52055fc) No inactive management server node found 2025-06-29 03:09:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1410bdae]) (logid:a52055fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5869603]) (logid:49c4e5c7) No inactive management server node found 2025-06-29 03:09:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5869603]) (logid:49c4e5c7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:37,990 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8a545356]) (logid:46f18697) HA health check task is running... 2025-06-29 03:09:38,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f77cfbac]) (logid:7e9c5be2) No inactive management server node found 2025-06-29 03:09:38,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f77cfbac]) (logid:7e9c5be2) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:09:39,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65725a78]) (logid:f2138f2f) No inactive management server node found 2025-06-29 03:09:39,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65725a78]) (logid:f2138f2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-01c77e0e]) (logid:ab71c993) Found 0 routers to update status. 2025-06-29 03:09:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-01c77e0e]) (logid:ab71c993) Found 0 VPC's to update Redundant State. 2025-06-29 03:09:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-01c77e0e]) (logid:ab71c993) Found 0 networks to update RvR status. 2025-06-29 03:09:41,102 DEBUG [o.a.c.s.o.B.BucketUsageTask] (Bucket-Usage-1:[ctx-e690048b]) (logid:077f8f11) Completed updating bucket usage for all object stores 2025-06-29 03:09:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-bd4947c3]) (logid:ed39b94d) DbCollector is running... 2025-06-29 03:09:41,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b432f29]) (logid:28dc7ddd) No inactive management server node found 2025-06-29 03:09:41,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b432f29]) (logid:28dc7ddd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:41,992 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f40717d2]) (logid:51772d24) HA health check task is running... 2025-06-29 03:09:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) ManagementServerCollector is running... 2025-06-29 03:09:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) DB properties were already loaded 2025-06-29 03:09:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:09:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Execution of process [4062958] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:09:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Exit value of process [4062958] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:09:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Process [4062958] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:09:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) usage install: null 2025-06-29 03:09:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Sessions active: 0 2025-06-29 03:09:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:09:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062961] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:09:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:09:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062966] for command [/bin/bash -c uname -r ]. 2025-06-29 03:09:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:09:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062967] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:09:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) system memory from /proc: 4887515136 2025-06-29 03:09:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:09:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062972] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:09:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) free memory from /proc: 497565696 2025-06-29 03:09:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:09:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062977] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:09:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) used memory from /proc: 768672 2025-06-29 03:09:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:09:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062978] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:09:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:09:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062979] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:09:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:09:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062980] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:09:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:09:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062987] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:09:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:09:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Successfully executed process [4062988] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail buffersdirect.count, 120 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail buffersdirect.used, 658750 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail buffersmapped.count, 7 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail buffersmapped.used, 798602 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail gcPS-Scavenge.count, 503 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail gcPS-Scavenge.time, 3663 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail jvmuptime, 43322323 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memoryheap.committed, 355467264 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memoryheap.init, 77594624 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memoryheap.usage, 0.1121985506991769 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorynon-heap.max, -1 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorynon-heap.usage, -2.20619912E8 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorynon-heap.used, 220619912 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1762540405891762 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.PS-Eden-Space.usage, 0.04265139977682828 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.PS-Old-Gen.usage, 0.1208175161914134 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorypools.PS-Survivor-Space.usage, 0.8738121619591346 2025-06-29 03:09:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorytotal.committed, 580517888 2025-06-29 03:09:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorytotal.init, 85262336 2025-06-29 03:09:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorytotal.max, 1908932607 2025-06-29 03:09:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail memorytotal.used, 434856616 2025-06-29 03:09:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail threadsdeadlocks, [] 2025-06-29 03:09:42,255 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:09:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail threadsnew.count, 0 2025-06-29 03:09:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:09:42,259 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-7c714406]) (logid:6a183d41) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:09:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":142027440,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43322245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":434059736,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214179472,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497565696,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.06],"systemCyclesUsage":[66369860,53810208,4588036119],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:09:42,259 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1443, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:09:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":142027440,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43322245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":434059736,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214179472,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497565696,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.06],"systemCyclesUsage":[66369860,53810208,4588036119],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:09:42,259 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:09:42,263 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:09:42,263 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1443, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:09:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":142027440,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43322245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":434059736,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214179472,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497565696,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.06],"systemCyclesUsage":[66369860,53810208,4588036119],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:09:42,263 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-323:[ctx-5d1027c9]) (logid:800993f5) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:09:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":355467264,"freeJvmMemoryBytes":142027440,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43322245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":434059736,"totalCommitted":580517888,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214179472,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":497565696,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.06],"systemCyclesUsage":[66369860,53810208,4588036119],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:09:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7519df6a]) (logid:f1844602) No inactive management server node found 2025-06-29 03:09:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7519df6a]) (logid:f1844602) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de92f5f9]) (logid:e3876c3e) No inactive management server node found 2025-06-29 03:09:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de92f5f9]) (logid:e3876c3e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:45,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8045: Seq 7-8045: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:09:45,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8045 2025-06-29 03:09:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:09:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:09:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d32b8d9]) (logid:c641c9ed) No inactive management server node found 2025-06-29 03:09:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d32b8d9]) (logid:c641c9ed) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:45,993 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-83ba6757]) (logid:ae8d0f0c) HA health check task is running... 2025-06-29 03:09:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d973d867]) (logid:6bfbd30e) VM scheduler.poll is being called at 2025-06-29T01:10:00+0000 2025-06-29 03:09:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d973d867]) (logid:6bfbd30e) Got 0 scheduled jobs to be executed at 2025-06-29T01:10:00+0000 2025-06-29 03:09:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d973d867]) (logid:6bfbd30e) Cleaned up 0 VM scheduled job entries 2025-06-29 03:09:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-877f539d]) (logid:9bb3821c) VM Operation Thread Running 2025-06-29 03:09:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9d3df0b2]) (logid:f49c2027) ClusterDRS.poll is being called at 2025-06-29T01:10:00+0000 2025-06-29 03:09:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9d3df0b2]) (logid:f49c2027) Removed 0 old drs migration plans 2025-06-29 03:09:47,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:47,212 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:47,213 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:09:47,216 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:09:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-073d6bc7]) (logid:1004221e) No inactive management server node found 2025-06-29 03:09:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-073d6bc7]) (logid:1004221e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:48,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64d1e18d]) (logid:849d5ff3) No inactive management server node found 2025-06-29 03:09:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64d1e18d]) (logid:849d5ff3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:49,995 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6601713a]) (logid:741b29a3) HA health check task is running... 2025-06-29 03:09:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-905c562f]) (logid:c0a8cb7c) No inactive management server node found 2025-06-29 03:09:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-905c562f]) (logid:c0a8cb7c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcf60814]) (logid:0a979f8a) No inactive management server node found 2025-06-29 03:09:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcf60814]) (logid:0a979f8a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daae670a]) (logid:4c6ad184) No inactive management server node found 2025-06-29 03:09:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daae670a]) (logid:4c6ad184) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:53,996 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-595e22fc]) (logid:5fec692c) HA health check task is running... 2025-06-29 03:09:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d3045f6]) (logid:8daf263b) No inactive management server node found 2025-06-29 03:09:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d3045f6]) (logid:8daf263b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:55,131 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8047: Seq 7-8047: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:09:55,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8047 2025-06-29 03:09:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b248da76]) (logid:de8421c5) No inactive management server node found 2025-06-29 03:09:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b248da76]) (logid:de8421c5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f7c06418]) (logid:70f865d0) Enabled non-edge zones available for scan: 1 2025-06-29 03:09:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f7c06418]) (logid:70f865d0) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:09:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f7c06418]) (logid:70f865d0) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:09:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3cd07abc]) (logid:b535b30d) Enabled non-edge zones available for scan: 1 2025-06-29 03:09:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3cd07abc]) (logid:b535b30d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:09:57,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3a6e365]) (logid:1d6e733b) No inactive management server node found 2025-06-29 03:09:57,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3a6e365]) (logid:1d6e733b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:09:57,918 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-9c111c13]) (logid:e872fdaf) AutoScaling Monitor is running... 2025-06-29 03:09:57,997 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-30684969]) (logid:4eb633f5) HA health check task is running... 2025-06-29 03:09:58,041 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-1c211a42]) (logid:e24fc236) Starting to remove all volume_stats rows older than [Sat Jun 28 15:09:58 CEST 2025]. 2025-06-29 03:09:58,041 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-1c211a42]) (logid:e24fc236) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:09:58 CEST 2025]. 2025-06-29 03:09:59,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dae0d01b]) (logid:e33d829c) No inactive management server node found 2025-06-29 03:09:59,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dae0d01b]) (logid:e33d829c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08c7f743]) (logid:0f172129) No inactive management server node found 2025-06-29 03:10:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08c7f743]) (logid:0f172129) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:01,998 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-faf19ecc]) (logid:6ddbb931) HA health check task is running... 2025-06-29 03:10:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2e082c7]) (logid:dbdeeab5) No inactive management server node found 2025-06-29 03:10:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2e082c7]) (logid:dbdeeab5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:03,232 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:10:03,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:10:03,237 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) Seq 2-3048092522799696710: Routed from 90520738888109 2025-06-29 03:10:03,291 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) Seq 1-3048092522799696710: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:10:03,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:10:03,327 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) Seq 2-1612007191621929998: Routed from 90520738888109 2025-06-29 03:10:03,379 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5046653b]) (logid:6e06abb1) Seq 2-1612007191621929998: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:10:03,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a394811]) (logid:08d85f16) No inactive management server node found 2025-06-29 03:10:03,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a394811]) (logid:08d85f16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:05,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8048: Seq 7-8048: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:10:05,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8048 2025-06-29 03:10:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cce3fef]) (logid:d8da6eef) No inactive management server node found 2025-06-29 03:10:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cce3fef]) (logid:d8da6eef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:06,000 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-44a93349]) (logid:8aef4605) HA health check task is running... 2025-06-29 03:10:06,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3d0c492]) (logid:be6a72b9) No inactive management server node found 2025-06-29 03:10:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3d0c492]) (logid:be6a72b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad766d3d]) (logid:646b4df7) No inactive management server node found 2025-06-29 03:10:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad766d3d]) (logid:646b4df7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:10:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cdcdeaa]) (logid:51ef6db5) No inactive management server node found 2025-06-29 03:10:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cdcdeaa]) (logid:51ef6db5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:10,002 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a7f4bd11]) (logid:fe6e1e86) HA health check task is running... 2025-06-29 03:10:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-db48ec8e]) (logid:375d2ad4) Found 0 routers to update status. 2025-06-29 03:10:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-db48ec8e]) (logid:375d2ad4) Found 0 VPC's to update Redundant State. 2025-06-29 03:10:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-db48ec8e]) (logid:375d2ad4) Found 0 networks to update RvR status. 2025-06-29 03:10:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c6f21a8]) (logid:93d6947a) No inactive management server node found 2025-06-29 03:10:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c6f21a8]) (logid:93d6947a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-173a2710]) (logid:632ea3c0) No inactive management server node found 2025-06-29 03:10:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-173a2710]) (logid:632ea3c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:13,631 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-0234e00a]) (logid:1eb1d41c) Starting to remove all vm_stats rows older than [Sat Jun 28 15:10:13 CEST 2025]. 2025-06-29 03:10:13,667 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-0234e00a]) (logid:1eb1d41c) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:10:13 CEST 2025]. 2025-06-29 03:10:14,003 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-dc848d0c]) (logid:41aa6fc2) HA health check task is running... 2025-06-29 03:10:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aff7dfe4]) (logid:edcd8ef0) No inactive management server node found 2025-06-29 03:10:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aff7dfe4]) (logid:edcd8ef0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:15,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8049: Seq 7-8049: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:10:15,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8049 2025-06-29 03:10:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b8abc8b]) (logid:be3fba74) No inactive management server node found 2025-06-29 03:10:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b8abc8b]) (logid:be3fba74) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-50ed6c01]) (logid:21806307) Resetting hosts suitable for reconnect 2025-06-29 03:10:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-50ed6c01]) (logid:21806307) Completed resetting hosts suitable for reconnect 2025-06-29 03:10:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-50ed6c01]) (logid:21806307) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:10:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-50ed6c01]) (logid:21806307) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:10:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-50ed6c01]) (logid:21806307) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:10:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-50ed6c01]) (logid:21806307) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:10:17,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-902b4b38]) (logid:4383b053) No inactive management server node found 2025-06-29 03:10:17,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-902b4b38]) (logid:4383b053) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:18,005 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0848abc0]) (logid:7d66a7a8) HA health check task is running... 2025-06-29 03:10:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c79a81c]) (logid:4fec24bd) No inactive management server node found 2025-06-29 03:10:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c79a81c]) (logid:4fec24bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da70c9ac]) (logid:93245cae) No inactive management server node found 2025-06-29 03:10:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da70c9ac]) (logid:93245cae) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:21,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0716ecfd]) (logid:da27a995) No inactive management server node found 2025-06-29 03:10:21,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0716ecfd]) (logid:da27a995) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:22,006 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8a2eff69]) (logid:1fb30783) HA health check task is running... 2025-06-29 03:10:23,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c73fddc3]) (logid:081dfea9) No inactive management server node found 2025-06-29 03:10:23,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c73fddc3]) (logid:081dfea9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:10:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-312e7390]) (logid:d01ed33c) No inactive management server node found 2025-06-29 03:10:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-312e7390]) (logid:d01ed33c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:25,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8050: Seq 7-8050: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:10:25,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8050 2025-06-29 03:10:26,008 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5c31f510]) (logid:82030685) HA health check task is running... 2025-06-29 03:10:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c84d551c]) (logid:22f51c6f) No inactive management server node found 2025-06-29 03:10:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c84d551c]) (logid:22f51c6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6cf45723]) (logid:0f47d7c5) Enabled non-edge zones available for scan: 1 2025-06-29 03:10:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6cf45723]) (logid:0f47d7c5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:10:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6cf45723]) (logid:0f47d7c5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:10:26,896 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f1a7457]) (logid:a81b6772) Enabled non-edge zones available for scan: 1 2025-06-29 03:10:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f1a7457]) (logid:a81b6772) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:10:27,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2d82f34]) (logid:b0328f5b) No inactive management server node found 2025-06-29 03:10:27,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2d82f34]) (logid:b0328f5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b79ff59a]) (logid:1df91723) No inactive management server node found 2025-06-29 03:10:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b79ff59a]) (logid:1df91723) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:30,009 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e9c83179]) (logid:09979403) HA health check task is running... 2025-06-29 03:10:30,185 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:10:30,186 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:10:30,186 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) Seq 2-3048092522799696711: Routed from 90520738888109 2025-06-29 03:10:30,229 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) Seq 1-3048092522799696711: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:10:30,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:10:30,232 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) Seq 2-1612007191621929999: Routed from 90520738888109 2025-06-29 03:10:30,235 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e221500f]) (logid:9359d601) Seq 2-1612007191621929999: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:10:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b905b2f6]) (logid:fe4bd7d3) No inactive management server node found 2025-06-29 03:10:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b905b2f6]) (logid:fe4bd7d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:32,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-758ee89c]) (logid:e071df45) No inactive management server node found 2025-06-29 03:10:32,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-758ee89c]) (logid:e071df45) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:33,467 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) StorageCollector is running... 2025-06-29 03:10:33,473 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:10:33,473 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:10:33,474 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:10:33,476 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 2-6717118844223095486: Routed from 90520738888109 2025-06-29 03:10:33,529 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 4-6717118844223095486: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:10:33,533 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:10:33,533 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:10:33,533 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:10:33,534 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 2-3048092522799696712: Routed from 90520738888109 2025-06-29 03:10:33,582 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 1-3048092522799696712: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:10:33,592 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:10:33,592 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:10:33,592 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:10:33,593 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 2-3048092522799696713: Routed from 90520738888109 2025-06-29 03:10:33,680 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 1-3048092522799696713: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:10:33,690 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:10:33,691 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:10:33,691 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:10:33,692 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 2-3048092522799696714: Routed from 90520738888109 2025-06-29 03:10:33,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40a41627]) (logid:38e3d027) No inactive management server node found 2025-06-29 03:10:33,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40a41627]) (logid:38e3d027) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:34,010 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-22d40fba]) (logid:a6e8b81e) HA health check task is running... 2025-06-29 03:10:34,027 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:10:34,028 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:10:34,030 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:10:34,032 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:10:34,783 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 1-3048092522799696714: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:10:34,786 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:10:34,786 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:10:34,786 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:10:34,787 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 2-1612007191621930000: Routed from 90520738888109 2025-06-29 03:10:35,131 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8051: Seq 7-8051: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:10:35,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8051 2025-06-29 03:10:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c63af6a]) (logid:a7e41147) No inactive management server node found 2025-06-29 03:10:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c63af6a]) (logid:a7e41147) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:35,847 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-eba72439]) (logid:9ac15490) Seq 2-1612007191621930000: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:10:36,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1366ec08]) (logid:e63d5604) No inactive management server node found 2025-06-29 03:10:36,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1366ec08]) (logid:e63d5604) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:38,012 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d0319242]) (logid:80132946) HA health check task is running... 2025-06-29 03:10:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b7f33cf]) (logid:7cec171b) No inactive management server node found 2025-06-29 03:10:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b7f33cf]) (logid:7cec171b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f414b9a]) (logid:4c7caa44) No inactive management server node found 2025-06-29 03:10:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f414b9a]) (logid:4c7caa44) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-461a5e51]) (logid:c39e6612) Found 0 routers to update status. 2025-06-29 03:10:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-461a5e51]) (logid:c39e6612) Found 0 VPC's to update Redundant State. 2025-06-29 03:10:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-461a5e51]) (logid:c39e6612) Found 0 networks to update RvR status. 2025-06-29 03:10:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-9dc1a3f7]) (logid:62e65629) DbCollector is running... 2025-06-29 03:10:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7005a94]) (logid:194fce0b) No inactive management server node found 2025-06-29 03:10:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7005a94]) (logid:194fce0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:42,013 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3331c6fa]) (logid:7171b216) HA health check task is running... 2025-06-29 03:10:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) ManagementServerCollector is running... 2025-06-29 03:10:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) DB properties were already loaded 2025-06-29 03:10:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:10:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Execution of process [4063003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:10:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Exit value of process [4063003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:10:42,170 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Process [4063003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:10:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) usage install: null 2025-06-29 03:10:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Sessions active: 0 2025-06-29 03:10:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:10:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063006] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:10:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:10:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063011] for command [/bin/bash -c uname -r ]. 2025-06-29 03:10:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:10:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063012] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:10:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) system memory from /proc: 4887515136 2025-06-29 03:10:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:10:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063017] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:10:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) free memory from /proc: 492433408 2025-06-29 03:10:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:10:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063022] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:10:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) used memory from /proc: 768672 2025-06-29 03:10:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:10:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063023] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:10:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:10:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063024] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:10:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:10:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063025] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:10:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:10:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063032] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:10:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:10:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Successfully executed process [4063033] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail buffersdirect.count, 120 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail buffersdirect.used, 658750 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail buffersmapped.count, 7 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail buffersmapped.used, 798602 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail gcPS-Scavenge.count, 504 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail gcPS-Scavenge.time, 3671 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:10:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail jvmuptime, 43382324 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memoryheap.committed, 356515840 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memoryheap.init, 77594624 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memoryheap.usage, 0.10951056895561187 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorynon-heap.max, -1 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorynon-heap.usage, -2.20619912E8 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorynon-heap.used, 220619912 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1762540405891762 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.PS-Eden-Space.usage, 0.035391816305458 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.PS-Old-Gen.usage, 0.12111502626098498 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorypools.PS-Survivor-Space.usage, 0.7767971583775112 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorytotal.committed, 581566464 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorytotal.init, 85262336 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorytotal.max, 1908932607 2025-06-29 03:10:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail memorytotal.used, 429726880 2025-06-29 03:10:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail threadsdeadlocks, [] 2025-06-29 03:10:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail threadsnew.count, 0 2025-06-29 03:10:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:10:42,250 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-e3c46406]) (logid:9d7ca5b1) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:10:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":148227432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43382247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428916688,"totalCommitted":581566464,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209056672,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492433408,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370185,53810518,4588059238],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:10:42,250 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1445, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:10:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":148227432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43382247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428916688,"totalCommitted":581566464,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209056672,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492433408,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370185,53810518,4588059238],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:10:42,250 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:10:42,254 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-323:[ctx-914eb0de]) (logid:681e08f8) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:10:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":148227432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43382247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428916688,"totalCommitted":581566464,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209056672,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492433408,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370185,53810518,4588059238],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:10:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:10:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1445, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:10:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":148227432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43382247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428916688,"totalCommitted":581566464,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":209056672,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492433408,"systemMemoryUsed":768672,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370185,53810518,4588059238],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:10:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c80f8480]) (logid:9db69367) No inactive management server node found 2025-06-29 03:10:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c80f8480]) (logid:9db69367) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:44,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca626e1b]) (logid:cb4008b0) No inactive management server node found 2025-06-29 03:10:44,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca626e1b]) (logid:cb4008b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:10:45,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8052: Seq 7-8052: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:10:45,153 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8052 2025-06-29 03:10:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:10:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:10:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49f925f3]) (logid:7e4113af) No inactive management server node found 2025-06-29 03:10:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49f925f3]) (logid:7e4113af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:46,015 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-db656511]) (logid:d31c3213) HA health check task is running... 2025-06-29 03:10:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-16422d9a]) (logid:808796b1) VM scheduler.poll is being called at 2025-06-29T01:11:00+0000 2025-06-29 03:10:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-16422d9a]) (logid:808796b1) Got 0 scheduled jobs to be executed at 2025-06-29T01:11:00+0000 2025-06-29 03:10:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-16422d9a]) (logid:808796b1) Cleaned up 0 VM scheduled job entries 2025-06-29 03:10:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-f702ef8d]) (logid:b2d4be0a) VM Operation Thread Running 2025-06-29 03:10:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-28238edb]) (logid:7a419e4c) ClusterDRS.poll is being called at 2025-06-29T01:11:00+0000 2025-06-29 03:10:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-28238edb]) (logid:7a419e4c) Removed 0 old drs migration plans 2025-06-29 03:10:47,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:10:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:10:47,232 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:10:47,236 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:10:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5735ebbd]) (logid:cd30e0e3) No inactive management server node found 2025-06-29 03:10:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5735ebbd]) (logid:cd30e0e3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:48,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4809e9f]) (logid:34884f1a) No inactive management server node found 2025-06-29 03:10:48,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4809e9f]) (logid:34884f1a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:50,016 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-75fd79f4]) (logid:883ccc9f) HA health check task is running... 2025-06-29 03:10:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e42854ac]) (logid:96b75c50) No inactive management server node found 2025-06-29 03:10:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e42854ac]) (logid:96b75c50) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:51,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c74a637]) (logid:cec5bb3b) No inactive management server node found 2025-06-29 03:10:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c74a637]) (logid:cec5bb3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:53,346 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4001ba5a]) (logid:006eecf1) No inactive management server node found 2025-06-29 03:10:53,346 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4001ba5a]) (logid:006eecf1) Peer scan is finished. profiler: Done. Duration: 14ms , profilerQueryActiveList: Done. Duration: 12ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:54,018 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a0c025ef]) (logid:582462fb) HA health check task is running... 2025-06-29 03:10:54,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d263df55]) (logid:ebaeb16e) No inactive management server node found 2025-06-29 03:10:54,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d263df55]) (logid:ebaeb16e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:55,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8054: Seq 7-8054: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:10:55,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8054 2025-06-29 03:10:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fd08aee]) (logid:e4e3e756) No inactive management server node found 2025-06-29 03:10:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fd08aee]) (logid:e4e3e756) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ed3964c3]) (logid:0a35165e) Enabled non-edge zones available for scan: 1 2025-06-29 03:10:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ed3964c3]) (logid:0a35165e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:10:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ed3964c3]) (logid:0a35165e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:10:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6f1d5b33]) (logid:daa24676) Enabled non-edge zones available for scan: 1 2025-06-29 03:10:56,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6f1d5b33]) (logid:daa24676) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:10:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a3d4a42]) (logid:90ae328c) No inactive management server node found 2025-06-29 03:10:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a3d4a42]) (logid:90ae328c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:10:57,920 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-796558e3]) (logid:4f128171) AutoScaling Monitor is running... 2025-06-29 03:10:58,019 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-549baa9d]) (logid:ff1f8aae) HA health check task is running... 2025-06-29 03:10:58,043 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-ec9345ec]) (logid:0fbd952d) Starting to remove all volume_stats rows older than [Sat Jun 28 15:10:58 CEST 2025]. 2025-06-29 03:10:58,044 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-ec9345ec]) (logid:0fbd952d) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:10:58 CEST 2025]. 2025-06-29 03:10:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89a561c4]) (logid:791b383c) No inactive management server node found 2025-06-29 03:10:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89a561c4]) (logid:791b383c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23ae6e83]) (logid:96409e97) No inactive management server node found 2025-06-29 03:11:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23ae6e83]) (logid:96409e97) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:02,020 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-07afcdf9]) (logid:7678a76f) HA health check task is running... 2025-06-29 03:11:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5001cfeb]) (logid:5ef1a581) No inactive management server node found 2025-06-29 03:11:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5001cfeb]) (logid:5ef1a581) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:03,433 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:11:03,437 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:11:03,438 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) Seq 2-3048092522799696715: Routed from 90520738888109 2025-06-29 03:11:03,451 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) Seq 1-3048092522799696715: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:11:03,501 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:11:03,502 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) Seq 2-1612007191621930001: Routed from 90520738888109 2025-06-29 03:11:03,558 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-682e8b9c]) (logid:02c464e4) Seq 2-1612007191621930001: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:11:03,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f43d97c]) (logid:d6848277) No inactive management server node found 2025-06-29 03:11:03,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f43d97c]) (logid:d6848277) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:05,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8055: Seq 7-8055: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:11:05,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8055 2025-06-29 03:11:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9b02a38]) (logid:b157056a) No inactive management server node found 2025-06-29 03:11:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9b02a38]) (logid:b157056a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:06,022 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6a250856]) (logid:8a9b3f7e) HA health check task is running... 2025-06-29 03:11:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c5abac9]) (logid:df3438b2) No inactive management server node found 2025-06-29 03:11:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c5abac9]) (logid:df3438b2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f26c0ae]) (logid:e944e6b6) No inactive management server node found 2025-06-29 03:11:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f26c0ae]) (logid:e944e6b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f28db6e]) (logid:8eaff3fc) No inactive management server node found 2025-06-29 03:11:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f28db6e]) (logid:8eaff3fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:10,023 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-80d610a4]) (logid:aec38e1b) HA health check task is running... 2025-06-29 03:11:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-63b0a9d1]) (logid:934ae401) Found 0 routers to update status. 2025-06-29 03:11:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-63b0a9d1]) (logid:934ae401) Found 0 VPC's to update Redundant State. 2025-06-29 03:11:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-63b0a9d1]) (logid:934ae401) Found 0 networks to update RvR status. 2025-06-29 03:11:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-757a996b]) (logid:c27526fa) No inactive management server node found 2025-06-29 03:11:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-757a996b]) (logid:c27526fa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b462cf1]) (logid:11c21245) No inactive management server node found 2025-06-29 03:11:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b462cf1]) (logid:11c21245) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:13,669 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-814e71ad]) (logid:b2f32ba7) Starting to remove all vm_stats rows older than [Sat Jun 28 15:11:13 CEST 2025]. 2025-06-29 03:11:13,702 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-814e71ad]) (logid:b2f32ba7) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:11:13 CEST 2025]. 2025-06-29 03:11:14,024 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1ec60b2e]) (logid:8e8ebb18) HA health check task is running... 2025-06-29 03:11:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-406b78c9]) (logid:c5b2395d) No inactive management server node found 2025-06-29 03:11:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-406b78c9]) (logid:c5b2395d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:15,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8056: Seq 7-8056: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:11:15,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8056 2025-06-29 03:11:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0879b292]) (logid:504c3daa) No inactive management server node found 2025-06-29 03:11:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0879b292]) (logid:504c3daa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:11:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74f01df]) (logid:ccef1a90) No inactive management server node found 2025-06-29 03:11:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74f01df]) (logid:ccef1a90) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:18,026 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d8de9869]) (logid:712a8785) HA health check task is running... 2025-06-29 03:11:18,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43ebf393]) (logid:f5b6a08e) No inactive management server node found 2025-06-29 03:11:18,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43ebf393]) (logid:f5b6a08e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36442196]) (logid:baaf3b29) No inactive management server node found 2025-06-29 03:11:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36442196]) (logid:baaf3b29) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28182523]) (logid:3e56ab57) No inactive management server node found 2025-06-29 03:11:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28182523]) (logid:3e56ab57) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:22,027 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f7721bea]) (logid:404f8cb3) HA health check task is running... 2025-06-29 03:11:23,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6299be1f]) (logid:ae64f8e9) No inactive management server node found 2025-06-29 03:11:23,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6299be1f]) (logid:ae64f8e9) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms 2025-06-29 03:11:24,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2217a703]) (logid:f6b70070) No inactive management server node found 2025-06-29 03:11:24,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2217a703]) (logid:f6b70070) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:25,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8057: Seq 7-8057: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:11:25,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8057 2025-06-29 03:11:26,028 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-40cb6d5e]) (logid:3ac1121c) HA health check task is running... 2025-06-29 03:11:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fca8375]) (logid:95ee3a26) No inactive management server node found 2025-06-29 03:11:26,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fca8375]) (logid:95ee3a26) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:11:26,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0dc5bd22]) (logid:86991a74) Enabled non-edge zones available for scan: 1 2025-06-29 03:11:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0dc5bd22]) (logid:86991a74) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:11:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0dc5bd22]) (logid:86991a74) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:11:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0a297437]) (logid:219774a0) Enabled non-edge zones available for scan: 1 2025-06-29 03:11:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0a297437]) (logid:219774a0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:11:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b18f963a]) (logid:385dcebc) No inactive management server node found 2025-06-29 03:11:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b18f963a]) (logid:385dcebc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c31e041d]) (logid:72f67408) No inactive management server node found 2025-06-29 03:11:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c31e041d]) (logid:72f67408) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:30,030 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-fb1de763]) (logid:8267f8b7) HA health check task is running... 2025-06-29 03:11:30,237 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:11:30,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:11:30,239 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) Seq 2-3048092522799696716: Routed from 90520738888109 2025-06-29 03:11:30,282 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) Seq 1-3048092522799696716: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:11:30,283 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:11:30,284 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) Seq 2-1612007191621930002: Routed from 90520738888109 2025-06-29 03:11:30,287 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f0c3dcfa]) (logid:9b4315a5) Seq 2-1612007191621930002: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:11:30,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a390303a]) (logid:d75b48ee) No inactive management server node found 2025-06-29 03:11:30,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a390303a]) (logid:d75b48ee) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9183fb4]) (logid:7368390c) No inactive management server node found 2025-06-29 03:11:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9183fb4]) (logid:7368390c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:33,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bf64835]) (logid:1dd65f69) No inactive management server node found 2025-06-29 03:11:33,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bf64835]) (logid:1dd65f69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:11:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:11:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:11:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:11:34,031 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f934d193]) (logid:94471536) HA health check task is running... 2025-06-29 03:11:35,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8058: Seq 7-8058: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:11:35,143 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8058 2025-06-29 03:11:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b1b6e58]) (logid:f88c37fd) No inactive management server node found 2025-06-29 03:11:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b1b6e58]) (logid:f88c37fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:35,847 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) StorageCollector is running... 2025-06-29 03:11:35,853 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:11:35,853 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:11:35,854 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:11:35,856 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-6717118844223095487: Routed from 90520738888109 2025-06-29 03:11:35,914 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 4-6717118844223095487: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:11:35,918 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:11:35,918 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:11:35,918 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:11:35,919 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-1612007191621930003: Routed from 90520738888109 2025-06-29 03:11:35,971 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-1612007191621930003: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:11:35,995 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:11:35,995 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:11:35,995 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:11:35,996 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-3048092522799696717: Routed from 90520738888109 2025-06-29 03:11:36,046 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 1-3048092522799696717: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:11:36,056 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:11:36,056 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:11:36,056 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:11:36,057 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-3048092522799696718: Routed from 90520738888109 2025-06-29 03:11:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f75ffdbf]) (logid:b483c900) No inactive management server node found 2025-06-29 03:11:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f75ffdbf]) (logid:b483c900) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:37,150 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 1-3048092522799696718: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:11:37,152 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:11:37,152 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:11:37,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:11:37,153 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-1612007191621930004: Routed from 90520738888109 2025-06-29 03:11:38,032 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5dc75380]) (logid:2f6493e8) HA health check task is running... 2025-06-29 03:11:38,216 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-dc19012a]) (logid:f1ec8c6a) Seq 2-1612007191621930004: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:11:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39fd0289]) (logid:0470b74b) No inactive management server node found 2025-06-29 03:11:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39fd0289]) (logid:0470b74b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:39,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63d269ff]) (logid:ba328a6e) No inactive management server node found 2025-06-29 03:11:39,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63d269ff]) (logid:ba328a6e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9da27904]) (logid:e2ebbb14) Found 0 routers to update status. 2025-06-29 03:11:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9da27904]) (logid:e2ebbb14) Found 0 VPC's to update Redundant State. 2025-06-29 03:11:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9da27904]) (logid:e2ebbb14) Found 0 networks to update RvR status. 2025-06-29 03:11:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-148d0d21]) (logid:92150a5c) DbCollector is running... 2025-06-29 03:11:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c644f38]) (logid:45af20a0) No inactive management server node found 2025-06-29 03:11:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c644f38]) (logid:45af20a0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:42,033 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f78ef882]) (logid:08702861) HA health check task is running... 2025-06-29 03:11:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) ManagementServerCollector is running... 2025-06-29 03:11:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) DB properties were already loaded 2025-06-29 03:11:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:11:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Execution of process [4063038] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:11:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Exit value of process [4063038] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:11:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Process [4063038] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:11:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) usage install: null 2025-06-29 03:11:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Sessions active: 0 2025-06-29 03:11:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:11:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063041] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:11:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:11:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063046] for command [/bin/bash -c uname -r ]. 2025-06-29 03:11:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:11:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063047] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:11:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) system memory from /proc: 4887515136 2025-06-29 03:11:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:11:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063052] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:11:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) free memory from /proc: 484216832 2025-06-29 03:11:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:11:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063057] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:11:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) used memory from /proc: 768800 2025-06-29 03:11:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:11:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063058] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:11:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:11:42,218 DEBUG [o.a.c.m.w.W.WebhookDeliveryCleanupWorker] (Webhook-Deliveries-Cleanup-Worker-1:[ctx-838577f7]) (logid:5fc0c428) Clearing old deliveries for webhooks with limit=10 using management server ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:11:42,220 DEBUG [o.a.c.m.w.W.WebhookDeliveryCleanupWorker] (Webhook-Deliveries-Cleanup-Worker-1:[ctx-838577f7]) (logid:5fc0c428) Cleared old deliveries with limit=10 for 0 webhooks 2025-06-29 03:11:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063059] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:11:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:11:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063060] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:11:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:11:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063067] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:11:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:11:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Successfully executed process [4063068] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail buffersdirect.count, 120 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail buffersdirect.used, 658750 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail buffersmapped.count, 7 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail buffersmapped.used, 798602 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail gcPS-Scavenge.count, 505 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail gcPS-Scavenge.time, 3681 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail jvmuptime, 43442326 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memoryheap.committed, 357040128 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memoryheap.init, 77594624 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memoryheap.usage, 0.10665349166689912 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:11:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorynon-heap.max, -1 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorynon-heap.usage, -2.20619912E8 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorynon-heap.used, 220619912 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1762540405891762 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.PS-Eden-Space.usage, 0.02729608088868683 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.PS-Old-Gen.usage, 0.12132671611818015 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorypools.PS-Survivor-Space.usage, 0.7381512872103987 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorytotal.committed, 582090752 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorytotal.init, 85262336 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorytotal.max, 1908932607 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail memorytotal.used, 424271632 2025-06-29 03:11:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail threadsdeadlocks, [] 2025-06-29 03:11:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail threadsnew.count, 0 2025-06-29 03:11:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:11:42,253 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-3c51e38a]) (logid:91f2bbb0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:11:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154435976,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43442246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423224064,"totalCommitted":582090752,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203594328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484216832,"systemMemoryUsed":768800,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370503,53810809,4588082371],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:11:42,253 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1447, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:11:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154435976,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43442246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423224064,"totalCommitted":582090752,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203594328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484216832,"systemMemoryUsed":768800,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370503,53810809,4588082371],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:11:42,253 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:11:42,258 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:11:42,258 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1447, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:11:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154435976,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43442246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423224064,"totalCommitted":582090752,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203594328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484216832,"systemMemoryUsed":768800,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370503,53810809,4588082371],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:11:42,258 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-324:[ctx-cbdaa9ab]) (logid:15462d7e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:11:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154435976,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43442246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423224064,"totalCommitted":582090752,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203594328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":484216832,"systemMemoryUsed":768800,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.05],"systemCyclesUsage":[66370503,53810809,4588082371],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:11:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6470cc2]) (logid:e38e13b6) No inactive management server node found 2025-06-29 03:11:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6470cc2]) (logid:e38e13b6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d735f6c]) (logid:87757f88) No inactive management server node found 2025-06-29 03:11:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d735f6c]) (logid:87757f88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:45,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8059: Seq 7-8059: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:11:45,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8059 2025-06-29 03:11:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:11:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:11:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be9a5d69]) (logid:948859ce) No inactive management server node found 2025-06-29 03:11:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be9a5d69]) (logid:948859ce) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:46,035 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bf554307]) (logid:7b90d1e3) HA health check task is running... 2025-06-29 03:11:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d2cf5db4]) (logid:7104f913) VM scheduler.poll is being called at 2025-06-29T01:12:00+0000 2025-06-29 03:11:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d2cf5db4]) (logid:7104f913) Got 0 scheduled jobs to be executed at 2025-06-29T01:12:00+0000 2025-06-29 03:11:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d2cf5db4]) (logid:7104f913) Cleaned up 0 VM scheduled job entries 2025-06-29 03:11:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-62c1a062]) (logid:63af14aa) VM Operation Thread Running 2025-06-29 03:11:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7dcaa4ae]) (logid:586a82c7) ClusterDRS.poll is being called at 2025-06-29T01:12:00+0000 2025-06-29 03:11:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7dcaa4ae]) (logid:586a82c7) Removed 0 old drs migration plans 2025-06-29 03:11:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8eda584a]) (logid:7e104eb9) Resetting hosts suitable for reconnect 2025-06-29 03:11:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8eda584a]) (logid:7e104eb9) Completed resetting hosts suitable for reconnect 2025-06-29 03:11:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8eda584a]) (logid:7e104eb9) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:11:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8eda584a]) (logid:7e104eb9) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:11:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8eda584a]) (logid:7e104eb9) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:11:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8eda584a]) (logid:7e104eb9) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:11:47,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:11:47,268 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:11:47,269 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:11:47,272 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:11:47,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef16f4da]) (logid:319475d4) No inactive management server node found 2025-06-29 03:11:47,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef16f4da]) (logid:319475d4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a755cf10]) (logid:fa3e0de4) No inactive management server node found 2025-06-29 03:11:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a755cf10]) (logid:fa3e0de4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:50,036 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-22b7f066]) (logid:80a30cb5) HA health check task is running... 2025-06-29 03:11:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d33a2ce3]) (logid:50d9125a) No inactive management server node found 2025-06-29 03:11:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d33a2ce3]) (logid:50d9125a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05cbb23b]) (logid:5236960f) No inactive management server node found 2025-06-29 03:11:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05cbb23b]) (logid:5236960f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:53,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03b670c8]) (logid:e80f0a60) No inactive management server node found 2025-06-29 03:11:53,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03b670c8]) (logid:e80f0a60) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:11:54,038 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-20908076]) (logid:5b27a22c) HA health check task is running... 2025-06-29 03:11:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffb4702f]) (logid:4dc7a23c) No inactive management server node found 2025-06-29 03:11:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffb4702f]) (logid:4dc7a23c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:55,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8061: Seq 7-8061: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:11:55,155 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8061 2025-06-29 03:11:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8ab33e4]) (logid:845bd705) No inactive management server node found 2025-06-29 03:11:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8ab33e4]) (logid:845bd705) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f859667c]) (logid:39bfb6e4) Enabled non-edge zones available for scan: 1 2025-06-29 03:11:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f859667c]) (logid:39bfb6e4) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:11:56,765 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f859667c]) (logid:39bfb6e4) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:11:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a8395c94]) (logid:81bcd5bd) Enabled non-edge zones available for scan: 1 2025-06-29 03:11:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a8395c94]) (logid:81bcd5bd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:11:57,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e87a4410]) (logid:be43a8c9) No inactive management server node found 2025-06-29 03:11:57,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e87a4410]) (logid:be43a8c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:11:57,922 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-c447e3e2]) (logid:a1fd7304) AutoScaling Monitor is running... 2025-06-29 03:11:58,039 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6cc84721]) (logid:3c46c280) HA health check task is running... 2025-06-29 03:11:58,045 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-d55d98d2]) (logid:644fb8b8) Starting to remove all volume_stats rows older than [Sat Jun 28 15:11:58 CEST 2025]. 2025-06-29 03:11:58,046 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-d55d98d2]) (logid:644fb8b8) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:11:58 CEST 2025]. 2025-06-29 03:11:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5576176e]) (logid:71e8ef0a) No inactive management server node found 2025-06-29 03:11:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5576176e]) (logid:71e8ef0a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:00,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7341d29e]) (logid:e8432507) No inactive management server node found 2025-06-29 03:12:00,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7341d29e]) (logid:e8432507) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:02,040 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f8f81a1a]) (logid:16771d86) HA health check task is running... 2025-06-29 03:12:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0174f4dd]) (logid:6c2be81a) No inactive management server node found 2025-06-29 03:12:02,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0174f4dd]) (logid:6c2be81a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:03,615 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:12:03,619 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:12:03,620 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) Seq 2-3048092522799696719: Routed from 90520738888109 2025-06-29 03:12:03,634 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) Seq 1-3048092522799696719: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:12:03,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:12:03,703 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) Seq 2-1612007191621930005: Routed from 90520738888109 2025-06-29 03:12:03,750 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-26d5122b]) (logid:1a125e34) Seq 2-1612007191621930005: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:12:03,824 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be1eb190]) (logid:19672851) No inactive management server node found 2025-06-29 03:12:03,824 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be1eb190]) (logid:19672851) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:05,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8062: Seq 7-8062: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:12:05,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8062 2025-06-29 03:12:05,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87b7e5dd]) (logid:5df596c4) No inactive management server node found 2025-06-29 03:12:05,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87b7e5dd]) (logid:5df596c4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:06,041 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-05ec24fb]) (logid:a381e01e) HA health check task is running... 2025-06-29 03:12:06,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65643b6c]) (logid:49ea235e) No inactive management server node found 2025-06-29 03:12:06,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65643b6c]) (logid:49ea235e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:08,330 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5503cfd8]) (logid:6ddcbad2) No inactive management server node found 2025-06-29 03:12:08,330 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5503cfd8]) (logid:6ddcbad2) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ec99391]) (logid:8e6f23cc) No inactive management server node found 2025-06-29 03:12:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ec99391]) (logid:8e6f23cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:10,043 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4d68f98b]) (logid:d3e73577) HA health check task is running... 2025-06-29 03:12:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7ab6c2c0]) (logid:c01adda5) Found 0 routers to update status. 2025-06-29 03:12:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7ab6c2c0]) (logid:c01adda5) Found 0 VPC's to update Redundant State. 2025-06-29 03:12:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7ab6c2c0]) (logid:c01adda5) Found 0 networks to update RvR status. 2025-06-29 03:12:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad04eac]) (logid:0946a5d8) No inactive management server node found 2025-06-29 03:12:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad04eac]) (logid:0946a5d8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bc014ed]) (logid:33d30c73) No inactive management server node found 2025-06-29 03:12:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bc014ed]) (logid:33d30c73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:13,704 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-2d48c217]) (logid:72d7dabb) Starting to remove all vm_stats rows older than [Sat Jun 28 15:12:13 CEST 2025]. 2025-06-29 03:12:13,726 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-2d48c217]) (logid:72d7dabb) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:12:13 CEST 2025]. 2025-06-29 03:12:14,044 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1b91516d]) (logid:dc092fe2) HA health check task is running... 2025-06-29 03:12:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3580a2b0]) (logid:a4146e81) No inactive management server node found 2025-06-29 03:12:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3580a2b0]) (logid:a4146e81) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:15,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8063: Seq 7-8063: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:12:15,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8063 2025-06-29 03:12:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1957769a]) (logid:84f224f3) No inactive management server node found 2025-06-29 03:12:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1957769a]) (logid:84f224f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44ae2452]) (logid:cae3ba4f) No inactive management server node found 2025-06-29 03:12:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44ae2452]) (logid:cae3ba4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:18,045 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4b678f08]) (logid:9b24b74b) HA health check task is running... 2025-06-29 03:12:18,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca8b4454]) (logid:15098c43) No inactive management server node found 2025-06-29 03:12:18,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca8b4454]) (logid:15098c43) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5dac7184]) (logid:aad4ac18) No inactive management server node found 2025-06-29 03:12:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5dac7184]) (logid:aad4ac18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca5f415a]) (logid:a541fc02) No inactive management server node found 2025-06-29 03:12:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca5f415a]) (logid:a541fc02) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:22,046 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-29d6c627]) (logid:bfa780ba) HA health check task is running... 2025-06-29 03:12:23,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a98954a7]) (logid:8ac7f341) No inactive management server node found 2025-06-29 03:12:23,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a98954a7]) (logid:8ac7f341) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e95f42e]) (logid:76ca2f53) No inactive management server node found 2025-06-29 03:12:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e95f42e]) (logid:76ca2f53) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:25,136 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8064: Seq 7-8064: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:12:25,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8064 2025-06-29 03:12:26,048 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-510f8641]) (logid:8b4cd6cf) HA health check task is running... 2025-06-29 03:12:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd78eabd]) (logid:15b01a31) No inactive management server node found 2025-06-29 03:12:26,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd78eabd]) (logid:15b01a31) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-db367873]) (logid:309b386e) Enabled non-edge zones available for scan: 1 2025-06-29 03:12:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-db367873]) (logid:309b386e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:12:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-db367873]) (logid:309b386e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:12:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ee80c87f]) (logid:e75730f0) Enabled non-edge zones available for scan: 1 2025-06-29 03:12:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ee80c87f]) (logid:e75730f0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:12:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49918495]) (logid:d8230ba4) No inactive management server node found 2025-06-29 03:12:27,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49918495]) (logid:d8230ba4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2c2ecce]) (logid:69c1d38f) No inactive management server node found 2025-06-29 03:12:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2c2ecce]) (logid:69c1d38f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:30,049 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c31151fd]) (logid:8b11b6c8) HA health check task is running... 2025-06-29 03:12:30,290 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:12:30,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:12:30,293 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) Seq 2-3048092522799696720: Routed from 90520738888109 2025-06-29 03:12:30,336 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) Seq 1-3048092522799696720: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:12:30,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:12:30,339 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) Seq 2-1612007191621930006: Routed from 90520738888109 2025-06-29 03:12:30,341 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9af8e098]) (logid:1ca59ab0) Seq 2-1612007191621930006: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:12:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64245860]) (logid:58f51353) No inactive management server node found 2025-06-29 03:12:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64245860]) (logid:58f51353) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83f9eb0c]) (logid:10f6dbd0) No inactive management server node found 2025-06-29 03:12:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83f9eb0c]) (logid:10f6dbd0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7deba4bd]) (logid:b3dd9d42) No inactive management server node found 2025-06-29 03:12:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7deba4bd]) (logid:b3dd9d42) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:12:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:12:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:12:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:12:34,050 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b5ec9fc2]) (logid:1aa93e3b) HA health check task is running... 2025-06-29 03:12:35,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8065: Seq 7-8065: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:12:35,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8065 2025-06-29 03:12:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4afde91]) (logid:5cda6a46) No inactive management server node found 2025-06-29 03:12:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4afde91]) (logid:5cda6a46) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e93aab80]) (logid:a8bf36d4) No inactive management server node found 2025-06-29 03:12:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e93aab80]) (logid:a8bf36d4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:38,052 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a9e27a10]) (logid:e9c77046) HA health check task is running... 2025-06-29 03:12:38,216 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) StorageCollector is running... 2025-06-29 03:12:38,223 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:12:38,223 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:12:38,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:12:38,224 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-6717118844223095488: Routed from 90520738888109 2025-06-29 03:12:38,277 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 4-6717118844223095488: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:12:38,281 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:12:38,281 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:12:38,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:12:38,282 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-1612007191621930007: Routed from 90520738888109 2025-06-29 03:12:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ea6bef0]) (logid:b3dea9b6) No inactive management server node found 2025-06-29 03:12:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ea6bef0]) (logid:b3dea9b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:38,334 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-1612007191621930007: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:12:38,347 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:12:38,347 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:12:38,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:12:38,348 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-1612007191621930008: Routed from 90520738888109 2025-06-29 03:12:38,440 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-1612007191621930008: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:12:38,451 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:12:38,451 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:12:38,451 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:12:38,452 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-3048092522799696721: Routed from 90520738888109 2025-06-29 03:12:39,502 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 1-3048092522799696721: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:12:39,505 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:12:39,505 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:12:39,505 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:12:39,506 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-1612007191621930009: Routed from 90520738888109 2025-06-29 03:12:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48486828]) (logid:a0540c30) No inactive management server node found 2025-06-29 03:12:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48486828]) (logid:a0540c30) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:40,565 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ecb5dee3]) (logid:d15c2956) Seq 2-1612007191621930009: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:12:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a280b64c]) (logid:aa467bf9) Found 0 routers to update status. 2025-06-29 03:12:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a280b64c]) (logid:aa467bf9) Found 0 VPC's to update Redundant State. 2025-06-29 03:12:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a280b64c]) (logid:aa467bf9) Found 0 networks to update RvR status. 2025-06-29 03:12:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-3b702ece]) (logid:2b12630f) DbCollector is running... 2025-06-29 03:12:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95370ebf]) (logid:4b260b59) No inactive management server node found 2025-06-29 03:12:41,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95370ebf]) (logid:4b260b59) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:42,053 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f1ba5427]) (logid:67dcb862) HA health check task is running... 2025-06-29 03:12:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) ManagementServerCollector is running... 2025-06-29 03:12:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) DB properties were already loaded 2025-06-29 03:12:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:12:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Execution of process [4063082] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:12:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Exit value of process [4063082] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:12:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Process [4063082] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:12:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) usage install: null 2025-06-29 03:12:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Sessions active: 0 2025-06-29 03:12:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:12:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063085] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:12:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:12:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063090] for command [/bin/bash -c uname -r ]. 2025-06-29 03:12:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:12:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063091] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:12:42,190 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) system memory from /proc: 4887515136 2025-06-29 03:12:42,190 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:12:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063096] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:12:42,196 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) free memory from /proc: 488456192 2025-06-29 03:12:42,196 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:12:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063101] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:12:42,212 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) used memory from /proc: 768928 2025-06-29 03:12:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:12:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063102] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:12:42,220 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:12:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063103] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:12:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:12:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063104] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:12:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:12:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063111] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:12:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:12:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Successfully executed process [4063112] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:12:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail buffersdirect.count, 120 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail buffersdirect.used, 658750 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail buffersmapped.count, 7 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail buffersmapped.used, 798602 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail gcPS-Scavenge.count, 506 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail gcPS-Scavenge.time, 3689 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail jvmuptime, 43502330 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memoryheap.committed, 357564416 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memoryheap.init, 77594624 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memoryheap.usage, 0.10384158726676222 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorynon-heap.committed, 225050624 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorynon-heap.max, -1 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorynon-heap.usage, -2.20619912E8 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorynon-heap.used, 220619912 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1762540405891762 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.PS-Eden-Space.usage, 0.01951310314894636 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.PS-Old-Gen.usage, 0.12189314069437134 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorypools.PS-Survivor-Space.usage, 0.6604258219401041 2025-06-29 03:12:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorytotal.committed, 582615040 2025-06-29 03:12:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorytotal.init, 85262336 2025-06-29 03:12:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorytotal.max, 1908932607 2025-06-29 03:12:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail memorytotal.used, 418888616 2025-06-29 03:12:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail threadsdeadlocks, [] 2025-06-29 03:12:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail threadsnew.count, 0 2025-06-29 03:12:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:12:42,258 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-d14696b1]) (logid:18b162a0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:12:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":160090320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43502246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":418094008,"totalCommitted":582615040,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":198226592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488456192,"systemMemoryUsed":768928,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66370817,53811123,4588105490],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:12:42,258 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1449, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:12:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":160090320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43502246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":418094008,"totalCommitted":582615040,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":198226592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488456192,"systemMemoryUsed":768928,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66370817,53811123,4588105490],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:12:42,258 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:12:42,263 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:12:42,264 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-325:[ctx-63842863]) (logid:3d6df705) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:12:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":160090320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43502246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":418094008,"totalCommitted":582615040,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":198226592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488456192,"systemMemoryUsed":768928,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66370817,53811123,4588105490],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:12:42,264 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1449, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:12:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":160090320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43502246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":418094008,"totalCommitted":582615040,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":198226592,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488456192,"systemMemoryUsed":768928,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66370817,53811123,4588105490],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:12:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbd50d19]) (logid:5cb3dc4b) No inactive management server node found 2025-06-29 03:12:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbd50d19]) (logid:5cb3dc4b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cd6a298]) (logid:30224faa) No inactive management server node found 2025-06-29 03:12:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cd6a298]) (logid:30224faa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:45,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8066: Seq 7-8066: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:12:45,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8066 2025-06-29 03:12:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:12:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:12:45,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3302eaf9]) (logid:ac799f4b) No inactive management server node found 2025-06-29 03:12:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3302eaf9]) (logid:ac799f4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1110fbc5]) (logid:2b0157b2) VM scheduler.poll is being called at 2025-06-29T01:13:00+0000 2025-06-29 03:12:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1110fbc5]) (logid:2b0157b2) Got 0 scheduled jobs to be executed at 2025-06-29T01:13:00+0000 2025-06-29 03:12:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1110fbc5]) (logid:2b0157b2) Cleaned up 0 VM scheduled job entries 2025-06-29 03:12:46,054 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8334b914]) (logid:449be9bc) HA health check task is running... 2025-06-29 03:12:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9e5e5337]) (logid:ebd4be58) VM Operation Thread Running 2025-06-29 03:12:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-29849472]) (logid:bf7cecf1) ClusterDRS.poll is being called at 2025-06-29T01:13:00+0000 2025-06-29 03:12:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-29849472]) (logid:bf7cecf1) Removed 0 old drs migration plans 2025-06-29 03:12:47,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:12:47,218 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:12:47,219 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:12:47,222 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:12:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-923cfa57]) (logid:4481f936) No inactive management server node found 2025-06-29 03:12:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-923cfa57]) (logid:4481f936) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:48,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88f7c05f]) (logid:5cec8af1) No inactive management server node found 2025-06-29 03:12:48,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88f7c05f]) (logid:5cec8af1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:50,055 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-dfeb1554]) (logid:158ef22a) HA health check task is running... 2025-06-29 03:12:50,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-846fc35d]) (logid:23cee4de) No inactive management server node found 2025-06-29 03:12:50,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-846fc35d]) (logid:23cee4de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:51,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-935c875d]) (logid:eb575057) No inactive management server node found 2025-06-29 03:12:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-935c875d]) (logid:eb575057) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:53,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5eb07ebf]) (logid:10361e37) No inactive management server node found 2025-06-29 03:12:53,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5eb07ebf]) (logid:10361e37) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:54,056 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9e7a3029]) (logid:0a36e183) HA health check task is running... 2025-06-29 03:12:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da5193c1]) (logid:2b0b049a) No inactive management server node found 2025-06-29 03:12:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da5193c1]) (logid:2b0b049a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:12:55,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8068: Seq 7-8068: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:12:55,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8068 2025-06-29 03:12:56,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c34feba8]) (logid:4f5e2e1d) No inactive management server node found 2025-06-29 03:12:56,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c34feba8]) (logid:4f5e2e1d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-50343505]) (logid:faef3501) Enabled non-edge zones available for scan: 1 2025-06-29 03:12:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-50343505]) (logid:faef3501) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:12:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-50343505]) (logid:faef3501) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:12:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b8150b1d]) (logid:23bd1afa) Enabled non-edge zones available for scan: 1 2025-06-29 03:12:56,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b8150b1d]) (logid:23bd1afa) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:12:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-004f7954]) (logid:d9b4f2e7) No inactive management server node found 2025-06-29 03:12:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-004f7954]) (logid:d9b4f2e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:12:57,924 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-2ace176a]) (logid:9c9dccf4) AutoScaling Monitor is running... 2025-06-29 03:12:58,048 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-1ed38e53]) (logid:3d5f0225) Starting to remove all volume_stats rows older than [Sat Jun 28 15:12:58 CEST 2025]. 2025-06-29 03:12:58,048 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-1ed38e53]) (logid:3d5f0225) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:12:58 CEST 2025]. 2025-06-29 03:12:58,058 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5bf17a68]) (logid:464f0f0b) HA health check task is running... 2025-06-29 03:12:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94ce0221]) (logid:3075bed8) No inactive management server node found 2025-06-29 03:12:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94ce0221]) (logid:3075bed8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b7cbad2]) (logid:c28a50f1) No inactive management server node found 2025-06-29 03:13:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b7cbad2]) (logid:c28a50f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:02,058 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f8b90cc8]) (logid:974539f8) HA health check task is running... 2025-06-29 03:13:02,338 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec82514c]) (logid:c729b81a) No inactive management server node found 2025-06-29 03:13:02,338 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec82514c]) (logid:c729b81a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:03,818 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:13:03,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b5185af]) (logid:a9fd6206) No inactive management server node found 2025-06-29 03:13:03,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b5185af]) (logid:a9fd6206) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:03,822 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:13:03,823 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) Seq 2-3048092522799696722: Routed from 90520738888109 2025-06-29 03:13:03,835 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) Seq 1-3048092522799696722: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:13:03,869 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:13:03,869 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) Seq 2-1612007191621930010: Routed from 90520738888109 2025-06-29 03:13:03,917 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9108c930]) (logid:3fe5f2d3) Seq 2-1612007191621930010: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:13:05,138 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8069: Seq 7-8069: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:13:05,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8069 2025-06-29 03:13:05,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51347f1a]) (logid:89c3cfbe) No inactive management server node found 2025-06-29 03:13:05,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51347f1a]) (logid:89c3cfbe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:06,060 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f73295a2]) (logid:e1c597fa) HA health check task is running... 2025-06-29 03:13:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbad0d3c]) (logid:cc242d16) No inactive management server node found 2025-06-29 03:13:06,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbad0d3c]) (logid:cc242d16) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac1dda45]) (logid:acb12842) No inactive management server node found 2025-06-29 03:13:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac1dda45]) (logid:acb12842) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-648e9488]) (logid:aa5e7c4d) No inactive management server node found 2025-06-29 03:13:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-648e9488]) (logid:aa5e7c4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:10,061 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-57097df1]) (logid:e39931cd) HA health check task is running... 2025-06-29 03:13:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ba8d396d]) (logid:d32ba216) Found 0 routers to update status. 2025-06-29 03:13:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ba8d396d]) (logid:d32ba216) Found 0 VPC's to update Redundant State. 2025-06-29 03:13:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ba8d396d]) (logid:d32ba216) Found 0 networks to update RvR status. 2025-06-29 03:13:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75ed4e7d]) (logid:df2680d0) No inactive management server node found 2025-06-29 03:13:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75ed4e7d]) (logid:df2680d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0415c25]) (logid:6a652935) No inactive management server node found 2025-06-29 03:13:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0415c25]) (logid:6a652935) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:13,728 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-2b492b9a]) (logid:79fd8c15) Starting to remove all vm_stats rows older than [Sat Jun 28 15:13:13 CEST 2025]. 2025-06-29 03:13:13,752 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-2b492b9a]) (logid:79fd8c15) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:13:13 CEST 2025]. 2025-06-29 03:13:14,063 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-efc2c2ea]) (logid:7a4b7fbd) HA health check task is running... 2025-06-29 03:13:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78963c0c]) (logid:bf32e640) No inactive management server node found 2025-06-29 03:13:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78963c0c]) (logid:bf32e640) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:15,138 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8070: Seq 7-8070: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:13:15,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8070 2025-06-29 03:13:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73869e75]) (logid:bc2f4208) No inactive management server node found 2025-06-29 03:13:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73869e75]) (logid:bc2f4208) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bb572259]) (logid:34c09573) Resetting hosts suitable for reconnect 2025-06-29 03:13:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bb572259]) (logid:34c09573) Completed resetting hosts suitable for reconnect 2025-06-29 03:13:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bb572259]) (logid:34c09573) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:13:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bb572259]) (logid:34c09573) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:13:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bb572259]) (logid:34c09573) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:13:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bb572259]) (logid:34c09573) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:13:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a83b39f]) (logid:df50bfe5) No inactive management server node found 2025-06-29 03:13:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a83b39f]) (logid:df50bfe5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:18,064 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-dae9e20e]) (logid:cae5cda4) HA health check task is running... 2025-06-29 03:13:18,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07ccbce2]) (logid:b0e7302e) No inactive management server node found 2025-06-29 03:13:18,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07ccbce2]) (logid:b0e7302e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e04bb758]) (logid:815e8b01) No inactive management server node found 2025-06-29 03:13:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e04bb758]) (logid:815e8b01) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:13:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a93edb7]) (logid:8afd7ffc) No inactive management server node found 2025-06-29 03:13:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a93edb7]) (logid:8afd7ffc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:13:22,065 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ec703635]) (logid:e8b141ec) HA health check task is running... 2025-06-29 03:13:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20c1b172]) (logid:98f3c17f) No inactive management server node found 2025-06-29 03:13:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20c1b172]) (logid:98f3c17f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ce44718]) (logid:cd948689) No inactive management server node found 2025-06-29 03:13:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ce44718]) (logid:cd948689) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:25,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8071: Seq 7-8071: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:13:25,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8071 2025-06-29 03:13:26,066 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-76920d82]) (logid:83d712a4) HA health check task is running... 2025-06-29 03:13:26,338 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69a2ca85]) (logid:9a6d2262) No inactive management server node found 2025-06-29 03:13:26,338 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69a2ca85]) (logid:9a6d2262) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-95759e41]) (logid:0e20aa6c) Enabled non-edge zones available for scan: 1 2025-06-29 03:13:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-95759e41]) (logid:0e20aa6c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:13:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-95759e41]) (logid:0e20aa6c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:13:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-72f7300d]) (logid:eb9806ec) Enabled non-edge zones available for scan: 1 2025-06-29 03:13:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-72f7300d]) (logid:eb9806ec) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:13:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4741b8d3]) (logid:1147d994) No inactive management server node found 2025-06-29 03:13:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4741b8d3]) (logid:1147d994) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73e5962b]) (logid:030ebc7b) No inactive management server node found 2025-06-29 03:13:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73e5962b]) (logid:030ebc7b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:30,068 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fa89498f]) (logid:e927ffcd) HA health check task is running... 2025-06-29 03:13:30,344 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:13:30,344 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:13:30,345 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) Seq 2-3048092522799696723: Routed from 90520738888109 2025-06-29 03:13:30,388 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) Seq 1-3048092522799696723: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:13:30,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:13:30,390 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) Seq 2-1612007191621930011: Routed from 90520738888109 2025-06-29 03:13:30,392 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f45d7441]) (logid:114d33b9) Seq 2-1612007191621930011: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:13:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e30581e]) (logid:da40b087) No inactive management server node found 2025-06-29 03:13:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e30581e]) (logid:da40b087) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e9d45d]) (logid:6c9047c5) No inactive management server node found 2025-06-29 03:13:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e9d45d]) (logid:6c9047c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aac4a807]) (logid:29c93b7f) No inactive management server node found 2025-06-29 03:13:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aac4a807]) (logid:29c93b7f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:13:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:13:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:13:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:13:34,069 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cb931216]) (logid:5c13e839) HA health check task is running... 2025-06-29 03:13:35,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8072: Seq 7-8072: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:13:35,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8072 2025-06-29 03:13:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb3e60b]) (logid:87acbbf6) No inactive management server node found 2025-06-29 03:13:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb3e60b]) (logid:87acbbf6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ca26359]) (logid:57c5a1c9) No inactive management server node found 2025-06-29 03:13:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ca26359]) (logid:57c5a1c9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:38,070 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b5299827]) (logid:c0f1b283) HA health check task is running... 2025-06-29 03:13:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94a89d69]) (logid:2553e4c0) No inactive management server node found 2025-06-29 03:13:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94a89d69]) (logid:2553e4c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c34e1e1]) (logid:a00a8e5e) No inactive management server node found 2025-06-29 03:13:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c34e1e1]) (logid:a00a8e5e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:40,566 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) StorageCollector is running... 2025-06-29 03:13:40,572 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:13:40,572 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:13:40,573 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:13:40,575 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-6717118844223095489: Routed from 90520738888109 2025-06-29 03:13:40,650 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 4-6717118844223095489: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:13:40,653 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:13:40,653 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:13:40,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:13:40,654 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-1612007191621930012: Routed from 90520738888109 2025-06-29 03:13:40,707 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-1612007191621930012: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:13:40,730 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:13:40,730 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:13:40,730 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:13:40,731 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-3048092522799696724: Routed from 90520738888109 2025-06-29 03:13:40,778 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 1-3048092522799696724: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:13:40,788 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:13:40,788 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:13:40,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:13:40,789 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-3048092522799696725: Routed from 90520738888109 2025-06-29 03:13:40,816 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-cf0d8e0e]) (logid:62c933a3) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:13:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-cf0d8e0e]) (logid:62c933a3) Snapshot scheduler is being called at [2025-06-29T01:13:40+0000]. 2025-06-29 03:13:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-cf0d8e0e]) (logid:62c933a3) There are [0] scheduled snapshots to be executed at [2025-06-29T01:13:40+0000]. 2025-06-29 03:13:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-0253efe7]) (logid:4ba6e16f) External devices are not used. Skipping external device usage collection 2025-06-29 03:13:40,979 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-da264d75]) (logid:255083df) Backup backup.poll is being called at 2025-06-29T01:13:40+0000 2025-06-29 03:13:40,997 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-9ce9108c]) (logid:b72db526) Found 0 running routers. 2025-06-29 03:13:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0d1e90c0]) (logid:262859f7) Found 0 routers to update status. 2025-06-29 03:13:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0d1e90c0]) (logid:262859f7) Found 0 VPC's to update Redundant State. 2025-06-29 03:13:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0d1e90c0]) (logid:262859f7) Found 0 networks to update RvR status. 2025-06-29 03:13:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-85380201]) (logid:a6b8c90a) DbCollector is running... 2025-06-29 03:13:41,283 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-752df0cf]) (logid:9bede1bd) Started resource counters recalculation periodic task. 2025-06-29 03:13:41,286 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-752df0cf]) (logid:9bede1bd) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:13:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3803135]) (logid:3e9d6ad1) No inactive management server node found 2025-06-29 03:13:41,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3803135]) (logid:3e9d6ad1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:41,605 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-752df0cf]) (logid:9bede1bd) Finished resource counters recalculation periodic task. 2025-06-29 03:13:41,881 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 1-3048092522799696725: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:13:41,883 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:13:41,883 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:13:41,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:13:41,884 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-1612007191621930013: Routed from 90520738888109 2025-06-29 03:13:42,071 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e28443b7]) (logid:de834738) HA health check task is running... 2025-06-29 03:13:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) ManagementServerCollector is running... 2025-06-29 03:13:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) DB properties were already loaded 2025-06-29 03:13:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:13:42,168 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Execution of process [4063118] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:13:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Exit value of process [4063118] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:13:42,168 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Process [4063118] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:13:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) usage install: null 2025-06-29 03:13:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Sessions active: 0 2025-06-29 03:13:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:13:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063121] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:13:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:13:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063126] for command [/bin/bash -c uname -r ]. 2025-06-29 03:13:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:13:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063127] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:13:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) system memory from /proc: 4887515136 2025-06-29 03:13:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:13:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063132] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:13:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) free memory from /proc: 486391808 2025-06-29 03:13:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:13:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063137] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:13:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) used memory from /proc: 769056 2025-06-29 03:13:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:13:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063138] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:13:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:13:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063139] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:13:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:13:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063140] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:13:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:13:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063147] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:13:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:13:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Successfully executed process [4063148] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:13:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:13:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail buffersdirect.count, 120 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail buffersdirect.used, 658750 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail buffersmapped.count, 7 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail buffersmapped.used, 798602 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail gcPS-Scavenge.count, 507 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail gcPS-Scavenge.time, 3697 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail jvmuptime, 43562323 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memoryheap.committed, 357040128 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memoryheap.init, 77594624 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memoryheap.usage, 0.10631829282471977 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorynon-heap.max, -1 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorynon-heap.usage, -2.20652168E8 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorynon-heap.used, 220652168 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.PS-Eden-Space.usage, 0.026374560248232034 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.PS-Old-Gen.usage, 0.12223073371990571 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorypools.PS-Survivor-Space.usage, 0.6741125924246651 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorytotal.committed, 582156288 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorytotal.init, 85262336 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorytotal.max, 1908932607 2025-06-29 03:13:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail memorytotal.used, 423662768 2025-06-29 03:13:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail threadsdeadlocks, [] 2025-06-29 03:13:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail threadsnew.count, 0 2025-06-29 03:13:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) not storing detail threadstimed_waiting.count, 86 2025-06-29 03:13:42,250 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-6c953ee7]) (logid:9bcd6801) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:13:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154832376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43562245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":422859920,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202954456,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":486391808,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66371166,53811444,4588128557],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:13:42,250 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1451, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:13:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154832376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43562245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":422859920,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202954456,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":486391808,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66371166,53811444,4588128557],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:13:42,250 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:13:42,253 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-325:[ctx-d08b121b]) (logid:cf1eafcb) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:13:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154832376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43562245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":422859920,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202954456,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":486391808,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66371166,53811444,4588128557],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:13:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:13:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1451, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:13:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":154832376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43562245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":422859920,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202954456,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":123,"systemMemoryTotal":4887515136,"systemMemoryFree":486391808,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.0,0.04],"systemCyclesUsage":[66371166,53811444,4588128557],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:13:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96d47583]) (logid:3c80c14f) No inactive management server node found 2025-06-29 03:13:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96d47583]) (logid:3c80c14f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:42,945 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-090a7ca1]) (logid:110416dc) Seq 2-1612007191621930013: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:13:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc172861]) (logid:fe8dd771) No inactive management server node found 2025-06-29 03:13:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc172861]) (logid:fe8dd771) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:45,140 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8073: Seq 7-8073: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:13:45,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8073 2025-06-29 03:13:45,339 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:13:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:13:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9220c063]) (logid:bd7d0c0d) No inactive management server node found 2025-06-29 03:13:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9220c063]) (logid:bd7d0c0d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d3a9f4a4]) (logid:3860d70b) VM scheduler.poll is being called at 2025-06-29T01:14:00+0000 2025-06-29 03:13:46,052 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d3a9f4a4]) (logid:3860d70b) Got 0 scheduled jobs to be executed at 2025-06-29T01:14:00+0000 2025-06-29 03:13:46,054 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d3a9f4a4]) (logid:3860d70b) Cleaned up 0 VM scheduled job entries 2025-06-29 03:13:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-832de1d8]) (logid:ee279dee) VM Operation Thread Running 2025-06-29 03:13:46,072 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3a0ffef5]) (logid:ede0007e) HA health check task is running... 2025-06-29 03:13:46,186 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-3:[ctx-d30e6011]) (logid:c871d076) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:13:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9717a252]) (logid:20c34f7c) ClusterDRS.poll is being called at 2025-06-29T01:14:00+0000 2025-06-29 03:13:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9717a252]) (logid:20c34f7c) Removed 0 old drs migration plans 2025-06-29 03:13:47,253 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:13:47,255 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:13:47,256 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:13:47,259 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:13:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6656d644]) (logid:b85b2ef0) No inactive management server node found 2025-06-29 03:13:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6656d644]) (logid:b85b2ef0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:48,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac682cbc]) (logid:f909f949) No inactive management server node found 2025-06-29 03:13:48,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac682cbc]) (logid:f909f949) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:50,073 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-27ef8e63]) (logid:813725bc) HA health check task is running... 2025-06-29 03:13:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a083ed1f]) (logid:27e34b9a) No inactive management server node found 2025-06-29 03:13:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a083ed1f]) (logid:27e34b9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:51,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-134ccecf]) (logid:e0e82b53) No inactive management server node found 2025-06-29 03:13:51,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-134ccecf]) (logid:e0e82b53) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a18e987f]) (logid:0f241dd9) No inactive management server node found 2025-06-29 03:13:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a18e987f]) (logid:0f241dd9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:54,075 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-97598850]) (logid:588bf59f) HA health check task is running... 2025-06-29 03:13:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-876343bd]) (logid:1854858d) No inactive management server node found 2025-06-29 03:13:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-876343bd]) (logid:1854858d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:55,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8075: Seq 7-8075: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:13:55,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8075 2025-06-29 03:13:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72f17d6f]) (logid:97fc2921) No inactive management server node found 2025-06-29 03:13:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72f17d6f]) (logid:97fc2921) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ddbf2ffe]) (logid:b0169cb5) Enabled non-edge zones available for scan: 1 2025-06-29 03:13:56,765 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ddbf2ffe]) (logid:b0169cb5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:13:56,766 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ddbf2ffe]) (logid:b0169cb5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:13:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-afad1a4b]) (logid:7f23b867) Enabled non-edge zones available for scan: 1 2025-06-29 03:13:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-afad1a4b]) (logid:7f23b867) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:13:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d5e51b2]) (logid:f67f9769) No inactive management server node found 2025-06-29 03:13:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d5e51b2]) (logid:f67f9769) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:13:57,927 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-4aeb40a3]) (logid:f8ec7cdc) AutoScaling Monitor is running... 2025-06-29 03:13:58,051 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-e31fe64c]) (logid:df73ecbd) Starting to remove all volume_stats rows older than [Sat Jun 28 15:13:58 CEST 2025]. 2025-06-29 03:13:58,051 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-e31fe64c]) (logid:df73ecbd) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:13:58 CEST 2025]. 2025-06-29 03:13:58,076 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4f719bcc]) (logid:ce9b28ff) HA health check task is running... 2025-06-29 03:13:59,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ac1fbe3]) (logid:40e46a01) No inactive management server node found 2025-06-29 03:13:59,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ac1fbe3]) (logid:40e46a01) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:14:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43ffabcb]) (logid:9d971ab3) No inactive management server node found 2025-06-29 03:14:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43ffabcb]) (logid:9d971ab3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:02,077 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e03fa4a7]) (logid:0cec1c81) HA health check task is running... 2025-06-29 03:14:02,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48409e48]) (logid:30492b7f) No inactive management server node found 2025-06-29 03:14:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48409e48]) (logid:30492b7f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:03,828 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4008a967]) (logid:37bb6c22) No inactive management server node found 2025-06-29 03:14:03,828 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4008a967]) (logid:37bb6c22) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:03,970 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:14:03,973 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:14:03,974 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) Seq 2-3048092522799696726: Routed from 90520738888109 2025-06-29 03:14:03,987 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) Seq 1-3048092522799696726: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:14:04,032 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:14:04,033 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) Seq 2-1612007191621930014: Routed from 90520738888109 2025-06-29 03:14:04,089 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-711a8235]) (logid:95873d46) Seq 2-1612007191621930014: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:14:05,141 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8076: Seq 7-8076: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:14:05,164 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8076 2025-06-29 03:14:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce79c1e3]) (logid:7241d96b) No inactive management server node found 2025-06-29 03:14:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce79c1e3]) (logid:7241d96b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:06,078 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d1372e38]) (logid:54eabedd) HA health check task is running... 2025-06-29 03:14:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-969852ca]) (logid:26445812) No inactive management server node found 2025-06-29 03:14:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-969852ca]) (logid:26445812) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994fd39f]) (logid:3e6de615) No inactive management server node found 2025-06-29 03:14:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994fd39f]) (logid:3e6de615) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-564adefc]) (logid:d2a4f2f2) No inactive management server node found 2025-06-29 03:14:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-564adefc]) (logid:d2a4f2f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:10,079 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5e92b558]) (logid:f1d04b22) HA health check task is running... 2025-06-29 03:14:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-49c15ec4]) (logid:be92e610) Found 0 routers to update status. 2025-06-29 03:14:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-49c15ec4]) (logid:be92e610) Found 0 VPC's to update Redundant State. 2025-06-29 03:14:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-49c15ec4]) (logid:be92e610) Found 0 networks to update RvR status. 2025-06-29 03:14:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Running Capacity Checker ... 2025-06-29 03:14:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) recalculating system capacity 2025-06-29 03:14:11,067 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Executing cpu/ram capacity update 2025-06-29 03:14:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:11,073 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:11,080 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:11,084 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:14:11,084 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:14:11,085 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:14:11,096 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:11,097 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:11,102 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:11,103 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:14:11,103 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:14:11,104 DEBUG [c.c.c.CapacityManagerImpl] (pool-318-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:14:11,112 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Done executing cpu/ram capacity update 2025-06-29 03:14:11,112 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Executing storage capacity update 2025-06-29 03:14:11,118 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:14:11,118 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:14:11,118 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:14:11,120 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:14:11,124 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:14:11,124 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:14:11,124 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:14:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:14:11,131 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:14:11,131 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:14:11,132 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:14:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:14:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:14:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:14:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:14:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-319-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:14:11,143 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Done executing storage capacity update 2025-06-29 03:14:11,143 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Executing capacity updates for public ip and Vlans 2025-06-29 03:14:11,163 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Done capacity updates for public ip and Vlans 2025-06-29 03:14:11,163 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Executing capacity updates for private ip 2025-06-29 03:14:11,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Done executing capacity updates for private ip 2025-06-29 03:14:11,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Done recalculating system capacity 2025-06-29 03:14:11,194 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-41a27022]) (logid:176b267f) Done running Capacity Checker ... 2025-06-29 03:14:11,259 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-1c1aca8e]) (logid:b4e8aba8) Checking if any host reservation can be released ... 2025-06-29 03:14:11,264 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-1c1aca8e]) (logid:b4e8aba8) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:11,266 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-1c1aca8e]) (logid:b4e8aba8) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:11,266 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-1c1aca8e]) (logid:b4e8aba8) Done running HostReservationReleaseChecker ... 2025-06-29 03:14:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a20c086e]) (logid:4687a7e5) No inactive management server node found 2025-06-29 03:14:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a20c086e]) (logid:4687a7e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a4c032f]) (logid:4fe38084) No inactive management server node found 2025-06-29 03:14:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a4c032f]) (logid:4fe38084) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:13,753 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-cd408772]) (logid:1dac63b2) Starting to remove all vm_stats rows older than [Sat Jun 28 15:14:13 CEST 2025]. 2025-06-29 03:14:13,780 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-cd408772]) (logid:1dac63b2) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:14:13 CEST 2025]. 2025-06-29 03:14:14,081 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c0784a64]) (logid:87ce86f5) HA health check task is running... 2025-06-29 03:14:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-465e021a]) (logid:89c7b1f0) No inactive management server node found 2025-06-29 03:14:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-465e021a]) (logid:89c7b1f0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:15,141 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8077: Seq 7-8077: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:14:15,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8077 2025-06-29 03:14:15,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06a8344f]) (logid:52adaac3) No inactive management server node found 2025-06-29 03:14:15,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06a8344f]) (logid:52adaac3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dd631c8]) (logid:24cef2fb) No inactive management server node found 2025-06-29 03:14:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dd631c8]) (logid:24cef2fb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:18,083 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-00f03299]) (logid:403b2702) HA health check task is running... 2025-06-29 03:14:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1bdeb79]) (logid:6bd2e520) No inactive management server node found 2025-06-29 03:14:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1bdeb79]) (logid:6bd2e520) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd9de691]) (logid:23ea42af) No inactive management server node found 2025-06-29 03:14:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd9de691]) (logid:23ea42af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:21,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df7e93ad]) (logid:1181ef85) No inactive management server node found 2025-06-29 03:14:21,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df7e93ad]) (logid:1181ef85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:22,084 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ad9cb9ae]) (logid:749e456e) HA health check task is running... 2025-06-29 03:14:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b07f8300]) (logid:af2fd84c) No inactive management server node found 2025-06-29 03:14:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b07f8300]) (logid:af2fd84c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8437ec3d]) (logid:7e3049bb) No inactive management server node found 2025-06-29 03:14:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8437ec3d]) (logid:7e3049bb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:25,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8078: Seq 7-8078: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:14:25,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8078 2025-06-29 03:14:26,086 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-56573580]) (logid:7e8b6ffc) HA health check task is running... 2025-06-29 03:14:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f644e6bb]) (logid:7f902cb8) No inactive management server node found 2025-06-29 03:14:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f644e6bb]) (logid:7f902cb8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-62d865b9]) (logid:6879238b) Enabled non-edge zones available for scan: 1 2025-06-29 03:14:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-62d865b9]) (logid:6879238b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:14:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-62d865b9]) (logid:6879238b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:14:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6b70e631]) (logid:5c482eb0) Enabled non-edge zones available for scan: 1 2025-06-29 03:14:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6b70e631]) (logid:5c482eb0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:14:27,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91c58d76]) (logid:cdc20c2e) No inactive management server node found 2025-06-29 03:14:27,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91c58d76]) (logid:cdc20c2e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e87a4857]) (logid:bc1279c2) No inactive management server node found 2025-06-29 03:14:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e87a4857]) (logid:bc1279c2) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:30,087 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f0269c4a]) (logid:67e67a1f) HA health check task is running... 2025-06-29 03:14:30,395 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:14:30,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:14:30,398 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) Seq 2-3048092522799696727: Routed from 90520738888109 2025-06-29 03:14:30,441 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) Seq 1-3048092522799696727: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:14:30,442 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:14:30,443 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) Seq 2-1612007191621930015: Routed from 90520738888109 2025-06-29 03:14:30,446 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-a0253f86]) (logid:75f6cdac) Seq 2-1612007191621930015: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:14:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bfb88e8]) (logid:dc5b0220) No inactive management server node found 2025-06-29 03:14:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bfb88e8]) (logid:dc5b0220) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd88c7f9]) (logid:b978189c) No inactive management server node found 2025-06-29 03:14:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd88c7f9]) (logid:b978189c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:33,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-008a04e3]) (logid:e18dc0b0) No inactive management server node found 2025-06-29 03:14:33,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-008a04e3]) (logid:e18dc0b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:14:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:14:34,088 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-62d2876e]) (logid:bc90b7de) HA health check task is running... 2025-06-29 03:14:35,142 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8079: Seq 7-8079: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:14:35,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8079 2025-06-29 03:14:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0feeca0f]) (logid:910fa655) No inactive management server node found 2025-06-29 03:14:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0feeca0f]) (logid:910fa655) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:36,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2fdc2463]) (logid:46e32ac3) No inactive management server node found 2025-06-29 03:14:36,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2fdc2463]) (logid:46e32ac3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:38,090 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b8e3a070]) (logid:de7006a2) HA health check task is running... 2025-06-29 03:14:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cde64554]) (logid:c0e5c395) No inactive management server node found 2025-06-29 03:14:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cde64554]) (logid:c0e5c395) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:39,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47a17422]) (logid:51ba2882) No inactive management server node found 2025-06-29 03:14:39,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47a17422]) (logid:51ba2882) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bcf0fe1b]) (logid:ccdf8ca9) Found 0 routers to update status. 2025-06-29 03:14:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bcf0fe1b]) (logid:ccdf8ca9) Found 0 VPC's to update Redundant State. 2025-06-29 03:14:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bcf0fe1b]) (logid:ccdf8ca9) Found 0 networks to update RvR status. 2025-06-29 03:14:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-18503106]) (logid:1bd1beee) DbCollector is running... 2025-06-29 03:14:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9ca0292]) (logid:7d51aff5) No inactive management server node found 2025-06-29 03:14:41,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9ca0292]) (logid:7d51aff5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:42,091 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7b38f294]) (logid:720b99af) HA health check task is running... 2025-06-29 03:14:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) ManagementServerCollector is running... 2025-06-29 03:14:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) DB properties were already loaded 2025-06-29 03:14:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:14:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Execution of process [4063165] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:14:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Exit value of process [4063165] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:14:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Process [4063165] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:14:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) usage install: null 2025-06-29 03:14:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Sessions active: 0 2025-06-29 03:14:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:14:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063168] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:14:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:14:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063173] for command [/bin/bash -c uname -r ]. 2025-06-29 03:14:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:14:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063174] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:14:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) system memory from /proc: 4887515136 2025-06-29 03:14:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:14:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063179] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:14:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) free memory from /proc: 481001472 2025-06-29 03:14:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:14:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063184] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:14:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) used memory from /proc: 769056 2025-06-29 03:14:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:14:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063185] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:14:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:14:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063186] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:14:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:14:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063187] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:14:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:14:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063194] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:14:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:14:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Successfully executed process [4063195] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail buffersdirect.count, 120 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail buffersdirect.used, 658750 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail buffersmapped.count, 7 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail buffersmapped.used, 798602 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail gcPS-Scavenge.count, 508 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail gcPS-Scavenge.time, 3704 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail jvmuptime, 43622319 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memoryheap.committed, 357564416 2025-06-29 03:14:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memoryheap.init, 77594624 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memoryheap.usage, 0.10629581743725969 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorynon-heap.max, -1 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorynon-heap.usage, -2.20652168E8 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorynon-heap.used, 220652168 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.PS-Eden-Space.usage, 0.02620078737218081 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.PS-Old-Gen.usage, 0.12250501757705225 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorypools.PS-Survivor-Space.usage, 0.6303905618601832 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorytotal.committed, 582680576 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorytotal.init, 85262336 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorytotal.max, 1908932607 2025-06-29 03:14:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail memorytotal.used, 423626640 2025-06-29 03:14:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail threadsdeadlocks, [] 2025-06-29 03:14:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail threadsnew.count, 0 2025-06-29 03:14:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:14:42,245 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-e057b76c]) (logid:1af2ec70) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:14:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":155388688,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43622247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":422827896,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202927256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":481001472,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.01,0.04],"systemCyclesUsage":[66371474,53811742,4588151693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:14:42,246 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1453, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:14:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":155388688,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43622247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":422827896,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202927256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":481001472,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.01,0.04],"systemCyclesUsage":[66371474,53811742,4588151693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:14:42,246 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:14:42,249 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-325:[ctx-72ceaa30]) (logid:f4ea04c2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:14:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":155388688,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43622247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":422827896,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202927256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":481001472,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.01,0.04],"systemCyclesUsage":[66371474,53811742,4588151693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:14:42,249 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:14:42,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1453, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:14:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":155388688,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43622247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":422827896,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":202927256,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":481001472,"systemMemoryUsed":769056,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.01,0.04],"systemCyclesUsage":[66371474,53811742,4588151693],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:14:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fd1c316]) (logid:d8b90d5d) No inactive management server node found 2025-06-29 03:14:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fd1c316]) (logid:d8b90d5d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:42,945 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) StorageCollector is running... 2025-06-29 03:14:42,950 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:14:42,950 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:14:42,950 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:14:42,951 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 2-6717118844223095490: Routed from 90520738888109 2025-06-29 03:14:43,003 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 4-6717118844223095490: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:14:43,007 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:14:43,007 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:14:43,008 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:14:43,009 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 2-3048092522799696728: Routed from 90520738888109 2025-06-29 03:14:43,080 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 1-3048092522799696728: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:14:43,091 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:14:43,091 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:14:43,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:14:43,092 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 2-3048092522799696729: Routed from 90520738888109 2025-06-29 03:14:43,181 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 1-3048092522799696729: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:14:43,192 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:14:43,192 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:14:43,192 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:14:43,193 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 2-3048092522799696730: Routed from 90520738888109 2025-06-29 03:14:44,284 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 1-3048092522799696730: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:14:44,286 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:14:44,286 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:14:44,286 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:14:44,287 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 2-1612007191621930016: Routed from 90520738888109 2025-06-29 03:14:44,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cce9d2f3]) (logid:37c3156c) No inactive management server node found 2025-06-29 03:14:44,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cce9d2f3]) (logid:37c3156c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:45,153 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8080: Seq 7-8080: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:14:45,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8080 2025-06-29 03:14:45,344 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:14:45,360 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-717868bb]) (logid:0f0172a0) Seq 2-1612007191621930016: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:14:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:14:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-529a4ba0]) (logid:98c34b9c) No inactive management server node found 2025-06-29 03:14:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-529a4ba0]) (logid:98c34b9c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c6ed337d]) (logid:eb25374c) VM scheduler.poll is being called at 2025-06-29T01:15:00+0000 2025-06-29 03:14:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c6ed337d]) (logid:eb25374c) Got 0 scheduled jobs to be executed at 2025-06-29T01:15:00+0000 2025-06-29 03:14:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c6ed337d]) (logid:eb25374c) Cleaned up 0 VM scheduled job entries 2025-06-29 03:14:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-0d81eebf]) (logid:a6a60ba2) VM Operation Thread Running 2025-06-29 03:14:46,092 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3f24e470]) (logid:0e5aa949) HA health check task is running... 2025-06-29 03:14:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-0135d533]) (logid:d5569b15) ClusterDRS.poll is being called at 2025-06-29T01:15:00+0000 2025-06-29 03:14:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-0135d533]) (logid:d5569b15) Removed 0 old drs migration plans 2025-06-29 03:14:47,053 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7839cb1d]) (logid:46f21c1d) Resetting hosts suitable for reconnect 2025-06-29 03:14:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7839cb1d]) (logid:46f21c1d) Completed resetting hosts suitable for reconnect 2025-06-29 03:14:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7839cb1d]) (logid:46f21c1d) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:14:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7839cb1d]) (logid:46f21c1d) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:14:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7839cb1d]) (logid:46f21c1d) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:14:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7839cb1d]) (logid:46f21c1d) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:14:47,229 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:47,231 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:47,232 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:14:47,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:14:47,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a69a8ca1]) (logid:a5f66e55) No inactive management server node found 2025-06-29 03:14:47,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a69a8ca1]) (logid:a5f66e55) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c803174]) (logid:a52f2b5e) No inactive management server node found 2025-06-29 03:14:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c803174]) (logid:a52f2b5e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:50,093 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-afc7820d]) (logid:738f4749) HA health check task is running... 2025-06-29 03:14:50,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8082: Seq 7-8082: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:14:50,164 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8082 2025-06-29 03:14:50,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d005d1c4]) (logid:b75ac2b3) No inactive management server node found 2025-06-29 03:14:50,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d005d1c4]) (logid:b75ac2b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bc6b0ea]) (logid:f22ef497) No inactive management server node found 2025-06-29 03:14:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bc6b0ea]) (logid:f22ef497) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:53,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bba3ae3f]) (logid:a5e21f13) No inactive management server node found 2025-06-29 03:14:53,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bba3ae3f]) (logid:a5e21f13) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:54,095 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8fcfa28d]) (logid:8ed00856) HA health check task is running... 2025-06-29 03:14:54,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d6063bb]) (logid:c18bf6e8) No inactive management server node found 2025-06-29 03:14:54,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d6063bb]) (logid:c18bf6e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf854621]) (logid:85c2ebc6) No inactive management server node found 2025-06-29 03:14:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf854621]) (logid:85c2ebc6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-50b5f235]) (logid:205cc31d) Enabled non-edge zones available for scan: 1 2025-06-29 03:14:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-50b5f235]) (logid:205cc31d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:14:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-50b5f235]) (logid:205cc31d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:14:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7ddfe292]) (logid:97db7421) Enabled non-edge zones available for scan: 1 2025-06-29 03:14:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7ddfe292]) (logid:97db7421) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:14:57,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38be218f]) (logid:29db52d5) No inactive management server node found 2025-06-29 03:14:57,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38be218f]) (logid:29db52d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:14:57,929 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-55a4cc99]) (logid:bd280b3d) AutoScaling Monitor is running... 2025-06-29 03:14:58,053 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2013607e]) (logid:aa7736d6) Starting to remove all volume_stats rows older than [Sat Jun 28 15:14:58 CEST 2025]. 2025-06-29 03:14:58,053 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2013607e]) (logid:aa7736d6) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:14:58 CEST 2025]. 2025-06-29 03:14:58,096 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f461de2b]) (logid:3c3e94be) HA health check task is running... 2025-06-29 03:14:59,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7156d3f]) (logid:a9dbe08a) No inactive management server node found 2025-06-29 03:14:59,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7156d3f]) (logid:a9dbe08a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:00,143 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8083: Seq 7-8083: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:00,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8083 2025-06-29 03:15:00,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a68df878]) (logid:ad1fdfa3) No inactive management server node found 2025-06-29 03:15:00,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a68df878]) (logid:ad1fdfa3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:02,097 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7edfbbc9]) (logid:ca80e288) HA health check task is running... 2025-06-29 03:15:02,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dfbc311]) (logid:a4bd6e32) No inactive management server node found 2025-06-29 03:15:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dfbc311]) (logid:a4bd6e32) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ade75be]) (logid:032a32f9) No inactive management server node found 2025-06-29 03:15:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ade75be]) (logid:032a32f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:04,146 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:15:04,150 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:15:04,152 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) Seq 2-3048092522799696731: Routed from 90520738888109 2025-06-29 03:15:04,166 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) Seq 1-3048092522799696731: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:15:04,216 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:15:04,219 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:15:04,219 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) Seq 2-1612007191621930017: Routed from 90520738888109 2025-06-29 03:15:04,265 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-df25e9f4]) (logid:2ecf50d2) Seq 2-1612007191621930017: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:15:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88a08003]) (logid:2ce2d1dc) No inactive management server node found 2025-06-29 03:15:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88a08003]) (logid:2ce2d1dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:06,099 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5e4c8aab]) (logid:cb4201ec) HA health check task is running... 2025-06-29 03:15:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b78e33b]) (logid:ef8a0f8f) No inactive management server node found 2025-06-29 03:15:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b78e33b]) (logid:ef8a0f8f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b11f9f05]) (logid:790afeb0) No inactive management server node found 2025-06-29 03:15:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b11f9f05]) (logid:790afeb0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2730887b]) (logid:95052322) No inactive management server node found 2025-06-29 03:15:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2730887b]) (logid:95052322) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:10,100 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1559ac7f]) (logid:4fadf9d5) HA health check task is running... 2025-06-29 03:15:10,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8084: Seq 7-8084: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:10,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8084 2025-06-29 03:15:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-54c1058c]) (logid:fe7037b3) Found 0 routers to update status. 2025-06-29 03:15:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-54c1058c]) (logid:fe7037b3) Found 0 VPC's to update Redundant State. 2025-06-29 03:15:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-54c1058c]) (logid:fe7037b3) Found 0 networks to update RvR status. 2025-06-29 03:15:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2738dd]) (logid:e5e0e913) No inactive management server node found 2025-06-29 03:15:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2738dd]) (logid:e5e0e913) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80dafb26]) (logid:c7fb0288) No inactive management server node found 2025-06-29 03:15:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80dafb26]) (logid:c7fb0288) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:13,782 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-55626c06]) (logid:4392ba0c) Starting to remove all vm_stats rows older than [Sat Jun 28 15:15:13 CEST 2025]. 2025-06-29 03:15:13,815 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-55626c06]) (logid:4392ba0c) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:15:13 CEST 2025]. 2025-06-29 03:15:14,101 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9b7ffda8]) (logid:46cea6a4) HA health check task is running... 2025-06-29 03:15:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f281748]) (logid:b72bc2b2) No inactive management server node found 2025-06-29 03:15:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f281748]) (logid:b72bc2b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1987b89a]) (logid:f9557334) No inactive management server node found 2025-06-29 03:15:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1987b89a]) (logid:f9557334) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1009be7b]) (logid:ba105abc) No inactive management server node found 2025-06-29 03:15:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1009be7b]) (logid:ba105abc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:15:18,102 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3f5dd8e6]) (logid:5a04d5d3) HA health check task is running... 2025-06-29 03:15:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95c6162d]) (logid:0487603f) No inactive management server node found 2025-06-29 03:15:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95c6162d]) (logid:0487603f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:20,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8085: Seq 7-8085: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:20,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8085 2025-06-29 03:15:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d38adf60]) (logid:6aadf8ee) No inactive management server node found 2025-06-29 03:15:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d38adf60]) (logid:6aadf8ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9eb14f34]) (logid:b9c9b01a) No inactive management server node found 2025-06-29 03:15:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9eb14f34]) (logid:b9c9b01a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:22,104 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cc622494]) (logid:c5d41785) HA health check task is running... 2025-06-29 03:15:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57fe7b88]) (logid:2bd8a548) No inactive management server node found 2025-06-29 03:15:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57fe7b88]) (logid:2bd8a548) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:15:24,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fdf58700]) (logid:53c1f0f4) No inactive management server node found 2025-06-29 03:15:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fdf58700]) (logid:53c1f0f4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:26,105 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-700b2692]) (logid:8f19a244) HA health check task is running... 2025-06-29 03:15:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7798c418]) (logid:ffe8bdae) No inactive management server node found 2025-06-29 03:15:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7798c418]) (logid:ffe8bdae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-33f5a3cf]) (logid:648a9ed2) Enabled non-edge zones available for scan: 1 2025-06-29 03:15:26,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-33f5a3cf]) (logid:648a9ed2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:15:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-33f5a3cf]) (logid:648a9ed2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:15:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-38ed04ab]) (logid:a02f552f) Enabled non-edge zones available for scan: 1 2025-06-29 03:15:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-38ed04ab]) (logid:a02f552f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:15:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad138408]) (logid:54323a00) No inactive management server node found 2025-06-29 03:15:27,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad138408]) (logid:54323a00) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:29,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b3e86d6]) (logid:af8b773d) No inactive management server node found 2025-06-29 03:15:29,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b3e86d6]) (logid:af8b773d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:30,106 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2e849f7c]) (logid:695fbdb3) HA health check task is running... 2025-06-29 03:15:30,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8086: Seq 7-8086: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:30,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8086 2025-06-29 03:15:30,448 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:15:30,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:15:30,449 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) Seq 2-3048092522799696732: Routed from 90520738888109 2025-06-29 03:15:30,493 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) Seq 1-3048092522799696732: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:15:30,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:15:30,496 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) Seq 2-1612007191621930018: Routed from 90520738888109 2025-06-29 03:15:30,500 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-86e6294b]) (logid:b8401413) Seq 2-1612007191621930018: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:15:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34f409eb]) (logid:04ffac3a) No inactive management server node found 2025-06-29 03:15:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34f409eb]) (logid:04ffac3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a89ac53]) (logid:2d1183ec) No inactive management server node found 2025-06-29 03:15:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a89ac53]) (logid:2d1183ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f231dccd]) (logid:da7c42aa) No inactive management server node found 2025-06-29 03:15:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f231dccd]) (logid:da7c42aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:15:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:15:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:15:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:15:34,108 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1d834031]) (logid:ce1c0f62) HA health check task is running... 2025-06-29 03:15:35,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8087: Seq 7-8087: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:35,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8087 2025-06-29 03:15:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a55818f]) (logid:f47d24d1) No inactive management server node found 2025-06-29 03:15:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a55818f]) (logid:f47d24d1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7397d6c]) (logid:d2e1a60f) No inactive management server node found 2025-06-29 03:15:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7397d6c]) (logid:d2e1a60f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:38,109 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7f5f4f30]) (logid:8aa97db2) HA health check task is running... 2025-06-29 03:15:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1d8964f]) (logid:fae54616) No inactive management server node found 2025-06-29 03:15:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1d8964f]) (logid:fae54616) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebb395c3]) (logid:fcc56046) No inactive management server node found 2025-06-29 03:15:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebb395c3]) (logid:fcc56046) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1cb210b0]) (logid:05469f12) Found 0 routers to update status. 2025-06-29 03:15:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1cb210b0]) (logid:05469f12) Found 0 VPC's to update Redundant State. 2025-06-29 03:15:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1cb210b0]) (logid:05469f12) Found 0 networks to update RvR status. 2025-06-29 03:15:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-e9e4508c]) (logid:3374f30a) DbCollector is running... 2025-06-29 03:15:41,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9758cb42]) (logid:a8d2a5fa) No inactive management server node found 2025-06-29 03:15:41,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9758cb42]) (logid:a8d2a5fa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:42,110 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-43ac24eb]) (logid:3a30fdb4) HA health check task is running... 2025-06-29 03:15:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) ManagementServerCollector is running... 2025-06-29 03:15:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) DB properties were already loaded 2025-06-29 03:15:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:15:42,167 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Execution of process [4063201] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:15:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Exit value of process [4063201] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:15:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Process [4063201] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:15:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) usage install: null 2025-06-29 03:15:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Sessions active: 0 2025-06-29 03:15:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:15:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063204] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:15:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:15:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063209] for command [/bin/bash -c uname -r ]. 2025-06-29 03:15:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:15:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063210] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:15:42,185 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) system memory from /proc: 4887515136 2025-06-29 03:15:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:15:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063215] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:15:42,191 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) free memory from /proc: 478420992 2025-06-29 03:15:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:15:42,203 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063220] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:15:42,203 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) used memory from /proc: 769184 2025-06-29 03:15:42,203 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:15:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063221] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:15:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:15:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063222] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:15:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:15:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063223] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:15:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:15:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063230] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:15:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:15:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Successfully executed process [4063231] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail buffersdirect.count, 120 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail buffersdirect.used, 658750 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail buffersmapped.count, 7 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail buffersmapped.used, 798602 2025-06-29 03:15:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail gcPS-Scavenge.count, 509 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail gcPS-Scavenge.time, 3712 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail jvmuptime, 43682319 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memoryheap.committed, 356515840 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memoryheap.init, 77594624 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memoryheap.usage, 0.10342488947624494 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorynon-heap.max, -1 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorynon-heap.usage, -2.20652168E8 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorynon-heap.used, 220652168 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.PS-Eden-Space.usage, 0.01806634809912705 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.PS-Old-Gen.usage, 0.12303805272686573 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorypools.PS-Survivor-Space.usage, 0.6502433189978967 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorytotal.committed, 581632000 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorytotal.init, 85262336 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorytotal.max, 1908932607 2025-06-29 03:15:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail memorytotal.used, 418127568 2025-06-29 03:15:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail threadsdeadlocks, [] 2025-06-29 03:15:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail threadsnew.count, 0 2025-06-29 03:15:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:15:42,247 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-569f0c43]) (logid:73da9176) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:15:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":159842312,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43682245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":417325696,"totalCommitted":581632000,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197431144,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478420992,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.02,0.04],"systemCyclesUsage":[66371789,53812054,4588174805],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:15:42,247 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1455, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:15:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":159842312,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43682245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":417325696,"totalCommitted":581632000,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197431144,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478420992,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.02,0.04],"systemCyclesUsage":[66371789,53812054,4588174805],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:15:42,247 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:15:42,253 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:15:42,253 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1455, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:15:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":159842312,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43682245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":417325696,"totalCommitted":581632000,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197431144,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478420992,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.02,0.04],"systemCyclesUsage":[66371789,53812054,4588174805],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:15:42,253 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-326:[ctx-82e2913e]) (logid:d38fa8a7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:15:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":356515840,"freeJvmMemoryBytes":159842312,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43682245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":417325696,"totalCommitted":581632000,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197431144,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478420992,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.02,0.04],"systemCyclesUsage":[66371789,53812054,4588174805],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:15:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccdf8ea9]) (logid:19ecd420) No inactive management server node found 2025-06-29 03:15:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccdf8ea9]) (logid:19ecd420) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:44,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dc9105e]) (logid:609271ea) No inactive management server node found 2025-06-29 03:15:44,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dc9105e]) (logid:609271ea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:45,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8088: Seq 7-8088: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:45,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8088 2025-06-29 03:15:45,361 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) StorageCollector is running... 2025-06-29 03:15:45,365 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:15:45,365 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:15:45,366 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:15:45,368 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 2-6717118844223095491: Routed from 90520738888109 2025-06-29 03:15:45,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:15:45,400 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 4-6717118844223095491: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:15:45,403 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:15:45,403 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:15:45,403 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:15:45,404 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 2-3048092522799696733: Routed from 90520738888109 2025-06-29 03:15:45,451 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 1-3048092522799696733: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:15:45,473 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:15:45,473 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:15:45,473 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:15:45,474 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 2-3048092522799696734: Routed from 90520738888109 2025-06-29 03:15:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:15:45,561 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 1-3048092522799696734: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:15:45,572 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:15:45,572 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:15:45,572 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:15:45,572 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 2-3048092522799696735: Routed from 90520738888109 2025-06-29 03:15:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5df7ac3]) (logid:f221f3d0) No inactive management server node found 2025-06-29 03:15:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5df7ac3]) (logid:f221f3d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-060aee66]) (logid:bbd75a33) VM scheduler.poll is being called at 2025-06-29T01:16:00+0000 2025-06-29 03:15:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-060aee66]) (logid:bbd75a33) Got 0 scheduled jobs to be executed at 2025-06-29T01:16:00+0000 2025-06-29 03:15:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-060aee66]) (logid:bbd75a33) Cleaned up 0 VM scheduled job entries 2025-06-29 03:15:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-4342b9af]) (logid:5d3d59ae) VM Operation Thread Running 2025-06-29 03:15:46,112 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-41539199]) (logid:b24005ef) HA health check task is running... 2025-06-29 03:15:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2b41c5d6]) (logid:ec67a7ff) ClusterDRS.poll is being called at 2025-06-29T01:16:00+0000 2025-06-29 03:15:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2b41c5d6]) (logid:ec67a7ff) Removed 0 old drs migration plans 2025-06-29 03:15:46,663 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 1-3048092522799696735: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:15:46,665 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:15:46,665 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:15:46,665 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:15:46,666 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 2-1612007191621930019: Routed from 90520738888109 2025-06-29 03:15:47,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:15:47,225 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:15:47,227 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:15:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:15:47,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42c485e7]) (logid:873ab574) No inactive management server node found 2025-06-29 03:15:47,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42c485e7]) (logid:873ab574) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:47,771 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1fb05cb3]) (logid:4a81f528) Seq 2-1612007191621930019: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:15:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a436af7]) (logid:8c263f2e) No inactive management server node found 2025-06-29 03:15:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a436af7]) (logid:8c263f2e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:50,113 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7146246c]) (logid:ba5a4ba8) HA health check task is running... 2025-06-29 03:15:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6f92c32]) (logid:e8961e3f) No inactive management server node found 2025-06-29 03:15:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6f92c32]) (logid:e8961e3f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:51,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d528e1fb]) (logid:393d389b) No inactive management server node found 2025-06-29 03:15:51,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d528e1fb]) (logid:393d389b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:53,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9347640a]) (logid:e93880e1) No inactive management server node found 2025-06-29 03:15:53,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9347640a]) (logid:e93880e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:54,114 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6eb192c4]) (logid:b839bc77) HA health check task is running... 2025-06-29 03:15:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b26963d]) (logid:d1cbcef6) No inactive management server node found 2025-06-29 03:15:54,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b26963d]) (logid:d1cbcef6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:15:55,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8090: Seq 7-8090: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:15:55,168 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8090 2025-06-29 03:15:56,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81f95bac]) (logid:cbc1a0f0) No inactive management server node found 2025-06-29 03:15:56,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81f95bac]) (logid:cbc1a0f0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:15:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a936ced1]) (logid:d7f63057) Enabled non-edge zones available for scan: 1 2025-06-29 03:15:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a936ced1]) (logid:d7f63057) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:15:56,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a936ced1]) (logid:d7f63057) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:15:56,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-16c8df24]) (logid:64ab881e) Enabled non-edge zones available for scan: 1 2025-06-29 03:15:56,897 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-16c8df24]) (logid:64ab881e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:15:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34e9d14d]) (logid:a9a9402f) No inactive management server node found 2025-06-29 03:15:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34e9d14d]) (logid:a9a9402f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:15:57,931 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-91b75c29]) (logid:8b6158c4) AutoScaling Monitor is running... 2025-06-29 03:15:58,056 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-1ed65784]) (logid:cd60646a) Starting to remove all volume_stats rows older than [Sat Jun 28 15:15:58 CEST 2025]. 2025-06-29 03:15:58,056 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-1ed65784]) (logid:cd60646a) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:15:58 CEST 2025]. 2025-06-29 03:15:58,115 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2355e230]) (logid:2e88f858) HA health check task is running... 2025-06-29 03:15:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b53785ef]) (logid:3554e69a) No inactive management server node found 2025-06-29 03:15:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b53785ef]) (logid:3554e69a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:00,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f3582b2]) (logid:acbb7d7b) No inactive management server node found 2025-06-29 03:16:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f3582b2]) (logid:acbb7d7b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:02,116 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5104bcbf]) (logid:11eb3ea4) HA health check task is running... 2025-06-29 03:16:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b865e43]) (logid:3df55f0f) No inactive management server node found 2025-06-29 03:16:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b865e43]) (logid:3df55f0f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dc95ec3]) (logid:ec5c9a79) No inactive management server node found 2025-06-29 03:16:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dc95ec3]) (logid:ec5c9a79) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:04,321 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:16:04,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:16:04,328 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) Seq 2-3048092522799696736: Routed from 90520738888109 2025-06-29 03:16:04,340 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) Seq 1-3048092522799696736: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:16:04,386 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:16:04,387 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) Seq 2-1612007191621930020: Routed from 90520738888109 2025-06-29 03:16:04,436 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-7adcde18]) (logid:2512b7ea) Seq 2-1612007191621930020: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:16:05,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8091: Seq 7-8091: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:16:05,167 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8091 2025-06-29 03:16:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17964370]) (logid:fc176e4d) No inactive management server node found 2025-06-29 03:16:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17964370]) (logid:fc176e4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:06,118 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-99e9799f]) (logid:305fbb8d) HA health check task is running... 2025-06-29 03:16:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0c5124b]) (logid:63e46d08) No inactive management server node found 2025-06-29 03:16:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0c5124b]) (logid:63e46d08) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c777ef3c]) (logid:8c286ac1) No inactive management server node found 2025-06-29 03:16:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c777ef3c]) (logid:8c286ac1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b77f936f]) (logid:89af9652) No inactive management server node found 2025-06-29 03:16:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b77f936f]) (logid:89af9652) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:10,119 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-06923403]) (logid:2b5be5a3) HA health check task is running... 2025-06-29 03:16:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e9f773e9]) (logid:522f1aee) Found 0 routers to update status. 2025-06-29 03:16:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e9f773e9]) (logid:522f1aee) Found 0 VPC's to update Redundant State. 2025-06-29 03:16:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e9f773e9]) (logid:522f1aee) Found 0 networks to update RvR status. 2025-06-29 03:16:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53b1da99]) (logid:40d6d688) No inactive management server node found 2025-06-29 03:16:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53b1da99]) (logid:40d6d688) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fdd9f750]) (logid:68ff5ee9) No inactive management server node found 2025-06-29 03:16:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fdd9f750]) (logid:68ff5ee9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:13,816 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-cdf42a99]) (logid:4d6eef7a) Starting to remove all vm_stats rows older than [Sat Jun 28 15:16:13 CEST 2025]. 2025-06-29 03:16:13,849 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-cdf42a99]) (logid:4d6eef7a) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:16:13 CEST 2025]. 2025-06-29 03:16:14,120 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f1c61c18]) (logid:acc750a4) HA health check task is running... 2025-06-29 03:16:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-805175a2]) (logid:90d162d2) No inactive management server node found 2025-06-29 03:16:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-805175a2]) (logid:90d162d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:15,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8092: Seq 7-8092: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:16:15,168 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8092 2025-06-29 03:16:15,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77d8fd2]) (logid:8688de99) No inactive management server node found 2025-06-29 03:16:15,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77d8fd2]) (logid:8688de99) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:17,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dcdd8b33]) (logid:ee9eb1b6) Resetting hosts suitable for reconnect 2025-06-29 03:16:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dcdd8b33]) (logid:ee9eb1b6) Completed resetting hosts suitable for reconnect 2025-06-29 03:16:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dcdd8b33]) (logid:ee9eb1b6) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:16:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dcdd8b33]) (logid:ee9eb1b6) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:16:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dcdd8b33]) (logid:ee9eb1b6) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:16:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dcdd8b33]) (logid:ee9eb1b6) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:16:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54dd0bb6]) (logid:4db3f6c3) No inactive management server node found 2025-06-29 03:16:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54dd0bb6]) (logid:4db3f6c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:18,122 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-dc406ace]) (logid:c3265772) HA health check task is running... 2025-06-29 03:16:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b0d7891]) (logid:d5b0ff14) No inactive management server node found 2025-06-29 03:16:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b0d7891]) (logid:d5b0ff14) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-075966e5]) (logid:7e63acd7) No inactive management server node found 2025-06-29 03:16:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-075966e5]) (logid:7e63acd7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:21,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df8916cc]) (logid:68e63969) No inactive management server node found 2025-06-29 03:16:21,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df8916cc]) (logid:68e63969) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:22,123 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9d16988c]) (logid:e852e632) HA health check task is running... 2025-06-29 03:16:23,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87616c18]) (logid:2bf58a31) No inactive management server node found 2025-06-29 03:16:23,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87616c18]) (logid:2bf58a31) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:24,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8da40284]) (logid:b8a5c45d) No inactive management server node found 2025-06-29 03:16:24,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8da40284]) (logid:b8a5c45d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:25,147 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8093: Seq 7-8093: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:16:25,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8093 2025-06-29 03:16:26,125 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e1a5b11a]) (logid:ed84d477) HA health check task is running... 2025-06-29 03:16:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a365301]) (logid:fd1fa44f) No inactive management server node found 2025-06-29 03:16:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a365301]) (logid:fd1fa44f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b37e7b05]) (logid:111d4681) Enabled non-edge zones available for scan: 1 2025-06-29 03:16:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b37e7b05]) (logid:111d4681) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:16:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b37e7b05]) (logid:111d4681) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:16:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9cdb5b15]) (logid:c1f72723) Enabled non-edge zones available for scan: 1 2025-06-29 03:16:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9cdb5b15]) (logid:c1f72723) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:16:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0697f9a2]) (logid:1962a387) No inactive management server node found 2025-06-29 03:16:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0697f9a2]) (logid:1962a387) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:29,343 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04dfea1a]) (logid:21f09fa6) No inactive management server node found 2025-06-29 03:16:29,343 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04dfea1a]) (logid:21f09fa6) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:30,126 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-732ead40]) (logid:98de0475) HA health check task is running... 2025-06-29 03:16:30,503 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:16:30,504 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:16:30,506 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) Seq 2-3048092522799696737: Routed from 90520738888109 2025-06-29 03:16:30,549 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) Seq 1-3048092522799696737: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:16:30,550 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:16:30,551 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) Seq 2-1612007191621930021: Routed from 90520738888109 2025-06-29 03:16:30,554 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c9588be4]) (logid:5f10cdde) Seq 2-1612007191621930021: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:16:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69c2704b]) (logid:66f3ea58) No inactive management server node found 2025-06-29 03:16:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69c2704b]) (logid:66f3ea58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a32d7f58]) (logid:453d4344) No inactive management server node found 2025-06-29 03:16:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a32d7f58]) (logid:453d4344) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7e95b84]) (logid:8c885126) No inactive management server node found 2025-06-29 03:16:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7e95b84]) (logid:8c885126) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:16:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:16:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:16:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:16:34,127 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-02688505]) (logid:e91549f4) HA health check task is running... 2025-06-29 03:16:35,147 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8094: Seq 7-8094: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:16:35,167 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8094 2025-06-29 03:16:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0624e00]) (logid:fea2de4a) No inactive management server node found 2025-06-29 03:16:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0624e00]) (logid:fea2de4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51ecf357]) (logid:d9725dd4) No inactive management server node found 2025-06-29 03:16:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51ecf357]) (logid:d9725dd4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:38,129 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2834ae56]) (logid:0e1c7cee) HA health check task is running... 2025-06-29 03:16:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5e4e87c]) (logid:a6066ee8) No inactive management server node found 2025-06-29 03:16:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5e4e87c]) (logid:a6066ee8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:39,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8699217]) (logid:1e93ffab) No inactive management server node found 2025-06-29 03:16:39,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8699217]) (logid:1e93ffab) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f74e6b6]) (logid:2dcb6490) Found 0 routers to update status. 2025-06-29 03:16:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f74e6b6]) (logid:2dcb6490) Found 0 VPC's to update Redundant State. 2025-06-29 03:16:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f74e6b6]) (logid:2dcb6490) Found 0 networks to update RvR status. 2025-06-29 03:16:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-e6a18459]) (logid:874da5a0) DbCollector is running... 2025-06-29 03:16:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-367067d9]) (logid:13509c55) No inactive management server node found 2025-06-29 03:16:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-367067d9]) (logid:13509c55) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:42,130 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7a8c07d0]) (logid:8d9c0d2d) HA health check task is running... 2025-06-29 03:16:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) ManagementServerCollector is running... 2025-06-29 03:16:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) DB properties were already loaded 2025-06-29 03:16:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:16:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Execution of process [4063245] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:16:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Exit value of process [4063245] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:16:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Process [4063245] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:16:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) usage install: null 2025-06-29 03:16:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Sessions active: 0 2025-06-29 03:16:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:16:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063248] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:16:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:16:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063253] for command [/bin/bash -c uname -r ]. 2025-06-29 03:16:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:16:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063254] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:16:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) system memory from /proc: 4887515136 2025-06-29 03:16:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:16:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063259] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:16:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) free memory from /proc: 478167040 2025-06-29 03:16:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:16:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063264] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:16:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) used memory from /proc: 769184 2025-06-29 03:16:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:16:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063265] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:16:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:16:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063266] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:16:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:16:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063267] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:16:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:16:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063274] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:16:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:16:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Successfully executed process [4063275] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail buffersdirect.count, 120 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail buffersdirect.used, 658750 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail buffersmapped.count, 7 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail buffersmapped.used, 798602 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail gcPS-Scavenge.count, 510 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail gcPS-Scavenge.time, 3718 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail jvmuptime, 43742324 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memoryheap.committed, 357040128 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memoryheap.init, 77594624 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memoryheap.usage, 0.10012376508160103 2025-06-29 03:16:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorynon-heap.max, -1 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorynon-heap.usage, -2.20652168E8 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorynon-heap.used, 220652168 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.PS-Eden-Space.usage, 0.008920548800563739 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.PS-Old-Gen.usage, 0.12348721202848945 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorypools.PS-Survivor-Space.usage, 0.581024169921875 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorytotal.committed, 582156288 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorytotal.init, 85262336 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorytotal.max, 1908932607 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail memorytotal.used, 411841016 2025-06-29 03:16:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail threadsdeadlocks, [] 2025-06-29 03:16:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail threadsnew.count, 0 2025-06-29 03:16:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:16:42,251 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-41006795]) (logid:8086a08e) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:16:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166662624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43742246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.32,"totalInit":85262336,"totalUsed":411029672,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191129520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478167040,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.32,0.1,0.06],"systemCyclesUsage":[66372105,53812369,4588197907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:16:42,251 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1457, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:16:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166662624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43742246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.32,"totalInit":85262336,"totalUsed":411029672,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191129520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478167040,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.32,0.1,0.06],"systemCyclesUsage":[66372105,53812369,4588197907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:16:42,251 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:16:42,257 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 03:16:42,257 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1457, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:16:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166662624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43742246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.32,"totalInit":85262336,"totalUsed":411029672,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191129520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478167040,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.32,0.1,0.06],"systemCyclesUsage":[66372105,53812369,4588197907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:16:42,257 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-327:[ctx-f63d0240]) (logid:74f2c8f2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:16:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166662624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43742246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.32,"totalInit":85262336,"totalUsed":411029672,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191129520,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":478167040,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.32,0.1,0.06],"systemCyclesUsage":[66372105,53812369,4588197907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:16:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ec44aa7]) (logid:d44a3d82) No inactive management server node found 2025-06-29 03:16:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ec44aa7]) (logid:d44a3d82) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b3e414d]) (logid:15ae0fd6) No inactive management server node found 2025-06-29 03:16:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b3e414d]) (logid:15ae0fd6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:45,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8095: Seq 7-8095: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:16:45,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8095 2025-06-29 03:16:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:16:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:16:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23858e00]) (logid:be168d9b) No inactive management server node found 2025-06-29 03:16:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23858e00]) (logid:be168d9b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-90dc5270]) (logid:c160d6da) VM scheduler.poll is being called at 2025-06-29T01:17:00+0000 2025-06-29 03:16:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-90dc5270]) (logid:c160d6da) Got 0 scheduled jobs to be executed at 2025-06-29T01:17:00+0000 2025-06-29 03:16:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-90dc5270]) (logid:c160d6da) Cleaned up 0 VM scheduled job entries 2025-06-29 03:16:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-4d2ebd58]) (logid:2ab6ec20) VM Operation Thread Running 2025-06-29 03:16:46,131 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-53601b79]) (logid:fc8443b0) HA health check task is running... 2025-06-29 03:16:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-989b3950]) (logid:4b57811a) ClusterDRS.poll is being called at 2025-06-29T01:17:00+0000 2025-06-29 03:16:46,338 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-989b3950]) (logid:4b57811a) Removed 0 old drs migration plans 2025-06-29 03:16:47,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:16:47,225 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:16:47,226 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:16:47,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:16:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3df027cd]) (logid:a3d24797) No inactive management server node found 2025-06-29 03:16:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3df027cd]) (logid:a3d24797) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:47,772 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) StorageCollector is running... 2025-06-29 03:16:47,778 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:16:47,778 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:16:47,778 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:16:47,778 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-6717118844223095492: Routed from 90520738888109 2025-06-29 03:16:47,832 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 4-6717118844223095492: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:16:47,837 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:16:47,837 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:16:47,837 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:16:47,838 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-1612007191621930022: Routed from 90520738888109 2025-06-29 03:16:47,889 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-1612007191621930022: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:16:47,913 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:16:47,913 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:16:47,913 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:16:47,914 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-3048092522799696738: Routed from 90520738888109 2025-06-29 03:16:47,961 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 1-3048092522799696738: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:16:47,971 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:16:47,971 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:16:47,971 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:16:47,972 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-3048092522799696739: Routed from 90520738888109 2025-06-29 03:16:48,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37f75211]) (logid:e2d8ee7f) No inactive management server node found 2025-06-29 03:16:48,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37f75211]) (logid:e2d8ee7f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:49,088 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 1-3048092522799696739: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:16:49,090 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:16:49,090 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:16:49,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:16:49,092 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-1612007191621930023: Routed from 90520738888109 2025-06-29 03:16:50,132 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1de409a3]) (logid:e20aae30) HA health check task is running... 2025-06-29 03:16:50,158 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-92234362]) (logid:a9d5c7bf) Seq 2-1612007191621930023: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:16:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0d36ba0]) (logid:c9a7d989) No inactive management server node found 2025-06-29 03:16:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0d36ba0]) (logid:c9a7d989) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58b68a82]) (logid:c70e6f14) No inactive management server node found 2025-06-29 03:16:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58b68a82]) (logid:c70e6f14) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6cfe420e]) (logid:1a56639e) No inactive management server node found 2025-06-29 03:16:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6cfe420e]) (logid:1a56639e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:54,134 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3121f75c]) (logid:d4447012) HA health check task is running... 2025-06-29 03:16:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55ec32e0]) (logid:b5bc0b9d) No inactive management server node found 2025-06-29 03:16:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55ec32e0]) (logid:b5bc0b9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:55,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8097: Seq 7-8097: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:16:55,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8097 2025-06-29 03:16:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-272a15cc]) (logid:d01bd5e2) No inactive management server node found 2025-06-29 03:16:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-272a15cc]) (logid:d01bd5e2) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-880b8684]) (logid:b18ab5d7) Enabled non-edge zones available for scan: 1 2025-06-29 03:16:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-880b8684]) (logid:b18ab5d7) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:16:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-880b8684]) (logid:b18ab5d7) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:16:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-540c6add]) (logid:5f4df16f) Enabled non-edge zones available for scan: 1 2025-06-29 03:16:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-540c6add]) (logid:5f4df16f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:16:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5696f36f]) (logid:ccc187f8) No inactive management server node found 2025-06-29 03:16:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5696f36f]) (logid:ccc187f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:16:57,933 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-617f1ed6]) (logid:01648c74) AutoScaling Monitor is running... 2025-06-29 03:16:58,059 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-8afda2f5]) (logid:3c1eb005) Starting to remove all volume_stats rows older than [Sat Jun 28 15:16:58 CEST 2025]. 2025-06-29 03:16:58,059 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-8afda2f5]) (logid:3c1eb005) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:16:58 CEST 2025]. 2025-06-29 03:16:58,135 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c4f14fe0]) (logid:44adf962) HA health check task is running... 2025-06-29 03:16:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18f0dba3]) (logid:4b61a2ea) No inactive management server node found 2025-06-29 03:16:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18f0dba3]) (logid:4b61a2ea) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:00,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8098: Seq 7-8098: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:00,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8098 2025-06-29 03:17:00,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eea6036e]) (logid:4e10599c) No inactive management server node found 2025-06-29 03:17:00,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eea6036e]) (logid:4e10599c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:02,137 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bc1bdf01]) (logid:9da6a7d8) HA health check task is running... 2025-06-29 03:17:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4597c41c]) (logid:70b6a0c6) No inactive management server node found 2025-06-29 03:17:02,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4597c41c]) (logid:70b6a0c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:03,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3358c3e5]) (logid:180a8f73) No inactive management server node found 2025-06-29 03:17:03,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3358c3e5]) (logid:180a8f73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:04,491 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:17:04,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:17:04,496 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) Seq 2-3048092522799696740: Routed from 90520738888109 2025-06-29 03:17:04,508 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) Seq 1-3048092522799696740: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:17:04,555 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:17:04,556 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) Seq 2-1612007191621930024: Routed from 90520738888109 2025-06-29 03:17:04,603 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-24703edc]) (logid:a673df3e) Seq 2-1612007191621930024: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:17:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994d39d8]) (logid:05c8792d) No inactive management server node found 2025-06-29 03:17:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994d39d8]) (logid:05c8792d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:06,138 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ea651c80]) (logid:83f3ec99) HA health check task is running... 2025-06-29 03:17:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b62f8318]) (logid:69e25d70) No inactive management server node found 2025-06-29 03:17:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b62f8318]) (logid:69e25d70) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c33b4cda]) (logid:dc8848fe) No inactive management server node found 2025-06-29 03:17:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c33b4cda]) (logid:dc8848fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:09,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06935b4f]) (logid:7dbad1d2) No inactive management server node found 2025-06-29 03:17:09,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06935b4f]) (logid:7dbad1d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:10,139 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-fb0ebe0c]) (logid:9f2f0bd8) HA health check task is running... 2025-06-29 03:17:10,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8099: Seq 7-8099: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:10,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8099 2025-06-29 03:17:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-919c74e5]) (logid:77886edb) Found 0 routers to update status. 2025-06-29 03:17:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-919c74e5]) (logid:77886edb) Found 0 VPC's to update Redundant State. 2025-06-29 03:17:11,005 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-919c74e5]) (logid:77886edb) Found 0 networks to update RvR status. 2025-06-29 03:17:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8cf8d10]) (logid:37d6aece) No inactive management server node found 2025-06-29 03:17:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8cf8d10]) (logid:37d6aece) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:12,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40e89327]) (logid:b1add63f) No inactive management server node found 2025-06-29 03:17:12,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40e89327]) (logid:b1add63f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:13,851 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-dc463895]) (logid:73a711bc) Starting to remove all vm_stats rows older than [Sat Jun 28 15:17:13 CEST 2025]. 2025-06-29 03:17:13,883 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-dc463895]) (logid:73a711bc) Removed a total of [5] vm_stats rows older than [Sat Jun 28 15:17:13 CEST 2025]. 2025-06-29 03:17:14,141 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1608bcb4]) (logid:f382f131) HA health check task is running... 2025-06-29 03:17:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3169e074]) (logid:8b39f781) No inactive management server node found 2025-06-29 03:17:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3169e074]) (logid:8b39f781) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-025fbcfc]) (logid:2072b7ee) No inactive management server node found 2025-06-29 03:17:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-025fbcfc]) (logid:2072b7ee) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14f13b24]) (logid:7e7f3b95) No inactive management server node found 2025-06-29 03:17:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14f13b24]) (logid:7e7f3b95) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:18,142 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-face2343]) (logid:d7ed1f42) HA health check task is running... 2025-06-29 03:17:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c538c1bd]) (logid:307d771c) No inactive management server node found 2025-06-29 03:17:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c538c1bd]) (logid:307d771c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:20,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8100: Seq 7-8100: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:20,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8100 2025-06-29 03:17:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fbca46c]) (logid:5d6c4cd3) No inactive management server node found 2025-06-29 03:17:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fbca46c]) (logid:5d6c4cd3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:17:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6c3b4ad]) (logid:88831860) No inactive management server node found 2025-06-29 03:17:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6c3b4ad]) (logid:88831860) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:22,143 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d64ab9f6]) (logid:19dc0695) HA health check task is running... 2025-06-29 03:17:23,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8163d8f4]) (logid:020ab01f) No inactive management server node found 2025-06-29 03:17:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8163d8f4]) (logid:020ab01f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9be3f9c7]) (logid:20b539e2) No inactive management server node found 2025-06-29 03:17:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9be3f9c7]) (logid:20b539e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:26,145 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-236a396d]) (logid:64c5f409) HA health check task is running... 2025-06-29 03:17:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df1e9f07]) (logid:9d6ef3aa) No inactive management server node found 2025-06-29 03:17:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df1e9f07]) (logid:9d6ef3aa) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-08f95b15]) (logid:68a4b6b3) Enabled non-edge zones available for scan: 1 2025-06-29 03:17:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-08f95b15]) (logid:68a4b6b3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:17:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-08f95b15]) (logid:68a4b6b3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:17:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-23ebbfc8]) (logid:2283f4f2) Enabled non-edge zones available for scan: 1 2025-06-29 03:17:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-23ebbfc8]) (logid:2283f4f2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:17:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-085fbc35]) (logid:7540ae36) No inactive management server node found 2025-06-29 03:17:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-085fbc35]) (logid:7540ae36) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eed4761b]) (logid:fdebadb7) No inactive management server node found 2025-06-29 03:17:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eed4761b]) (logid:fdebadb7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:30,146 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-14cc5125]) (logid:e9ff2bd9) HA health check task is running... 2025-06-29 03:17:30,150 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8101: Seq 7-8101: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:30,171 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8101 2025-06-29 03:17:30,557 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:17:30,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:17:30,559 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) Seq 2-3048092522799696741: Routed from 90520738888109 2025-06-29 03:17:30,602 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) Seq 1-3048092522799696741: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:17:30,603 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:17:30,604 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) Seq 2-1612007191621930025: Routed from 90520738888109 2025-06-29 03:17:30,607 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-e13c6fbf]) (logid:54e09f96) Seq 2-1612007191621930025: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:17:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea1a3518]) (logid:df4c16a9) No inactive management server node found 2025-06-29 03:17:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea1a3518]) (logid:df4c16a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83e5ee8a]) (logid:1eb040d7) No inactive management server node found 2025-06-29 03:17:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83e5ee8a]) (logid:1eb040d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23c118e9]) (logid:a48a9ff6) No inactive management server node found 2025-06-29 03:17:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23c118e9]) (logid:a48a9ff6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:17:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:17:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:17:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:17:34,148 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-30b86385]) (logid:f2992056) HA health check task is running... 2025-06-29 03:17:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2024e76c]) (logid:bada7fda) No inactive management server node found 2025-06-29 03:17:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2024e76c]) (logid:bada7fda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cefb5cb]) (logid:6374d91c) No inactive management server node found 2025-06-29 03:17:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cefb5cb]) (logid:6374d91c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:38,149 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0b7e3236]) (logid:ccf71827) HA health check task is running... 2025-06-29 03:17:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fd872c4]) (logid:2c3b29ac) No inactive management server node found 2025-06-29 03:17:38,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fd872c4]) (logid:2c3b29ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fd93047]) (logid:419bfddb) No inactive management server node found 2025-06-29 03:17:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fd93047]) (logid:419bfddb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:40,150 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8102: Seq 7-8102: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:40,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8102 2025-06-29 03:17:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8ed138f]) (logid:d387a158) Found 0 routers to update status. 2025-06-29 03:17:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8ed138f]) (logid:d387a158) Found 0 VPC's to update Redundant State. 2025-06-29 03:17:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8ed138f]) (logid:d387a158) Found 0 networks to update RvR status. 2025-06-29 03:17:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-3b699281]) (logid:cfca533f) DbCollector is running... 2025-06-29 03:17:41,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-081c3fca]) (logid:3fce81c0) No inactive management server node found 2025-06-29 03:17:41,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-081c3fca]) (logid:3fce81c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:42,150 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bdbcd57d]) (logid:a6f013e5) HA health check task is running... 2025-06-29 03:17:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) ManagementServerCollector is running... 2025-06-29 03:17:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) DB properties were already loaded 2025-06-29 03:17:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:17:42,171 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Execution of process [4063318] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:17:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Exit value of process [4063318] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:17:42,172 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Process [4063318] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:17:42,172 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) usage install: null 2025-06-29 03:17:42,172 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Sessions active: 0 2025-06-29 03:17:42,174 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:17:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063321] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:17:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:17:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063326] for command [/bin/bash -c uname -r ]. 2025-06-29 03:17:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:17:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063327] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:17:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) system memory from /proc: 4887515136 2025-06-29 03:17:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:17:42,201 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063332] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:17:42,202 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) free memory from /proc: 460275712 2025-06-29 03:17:42,202 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:17:42,221 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063337] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:17:42,221 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) used memory from /proc: 769184 2025-06-29 03:17:42,221 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:17:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063338] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:17:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:17:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063339] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:17:42,249 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:17:42,257 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063340] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:17:42,257 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:17:42,264 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063347] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:17:42,264 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:17:42,270 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Successfully executed process [4063348] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:17:42,272 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail buffersdirect.count, 120 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail buffersdirect.used, 658750 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail buffersmapped.count, 7 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail buffersmapped.used, 798602 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail gcPS-Scavenge.count, 510 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail gcPS-Scavenge.time, 3718 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail jvmuptime, 43802350 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memoryheap.committed, 357040128 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memoryheap.init, 77594624 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memoryheap.usage, 0.11241049741657512 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorynon-heap.max, -1 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorynon-heap.usage, -2.20652168E8 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorynon-heap.used, 220652168 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.PS-Eden-Space.usage, 0.0430250480646036 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.PS-Old-Gen.usage, 0.12348721202848945 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorypools.PS-Survivor-Space.usage, 0.581024169921875 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorytotal.committed, 582156288 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorytotal.init, 85262336 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorytotal.max, 1908932607 2025-06-29 03:17:42,273 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail memorytotal.used, 435289128 2025-06-29 03:17:42,274 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail threadsdeadlocks, [] 2025-06-29 03:17:42,274 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail threadsnew.count, 0 2025-06-29 03:17:42,275 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:17:42,277 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-1bbb933b]) (logid:c222ee78) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:17:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":143203056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43802249,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":434489240,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214584064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":460275712,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.17,0.09],"systemCyclesUsage":[66372534,53812684,4588220909],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:17:42,277 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1459, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:17:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":143203056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43802249,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":434489240,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214584064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":460275712,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.17,0.09],"systemCyclesUsage":[66372534,53812684,4588220909],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:17:42,277 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:17:42,290 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 13 ms 2025-06-29 03:17:42,290 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 13ms. agent: 0, pdu seq: 1459, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:17:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":143203056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43802249,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":434489240,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214584064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":460275712,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.17,0.09],"systemCyclesUsage":[66372534,53812684,4588220909],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:17:42,291 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-328:[ctx-08d5dac2]) (logid:e5e175af) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:17:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":143203056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43802249,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.4,"totalInit":85262336,"totalUsed":434489240,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":214584064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":460275712,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.4,0.17,0.09],"systemCyclesUsage":[66372534,53812684,4588220909],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:17:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03bd874c]) (logid:0fb35f25) No inactive management server node found 2025-06-29 03:17:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03bd874c]) (logid:0fb35f25) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a92b61e]) (logid:d763bc56) No inactive management server node found 2025-06-29 03:17:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a92b61e]) (logid:d763bc56) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:45,150 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8103: Seq 7-8103: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:45,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8103 2025-06-29 03:17:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:17:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:17:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f61ab7dc]) (logid:d991d3af) No inactive management server node found 2025-06-29 03:17:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f61ab7dc]) (logid:d991d3af) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d8d2162f]) (logid:05832c50) VM scheduler.poll is being called at 2025-06-29T01:18:00+0000 2025-06-29 03:17:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d8d2162f]) (logid:05832c50) Got 0 scheduled jobs to be executed at 2025-06-29T01:18:00+0000 2025-06-29 03:17:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d8d2162f]) (logid:05832c50) Cleaned up 0 VM scheduled job entries 2025-06-29 03:17:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-83bf5bd2]) (logid:fccb2936) VM Operation Thread Running 2025-06-29 03:17:46,152 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6aa123a1]) (logid:4394f7e7) HA health check task is running... 2025-06-29 03:17:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f885fe12]) (logid:12d3eb6e) ClusterDRS.poll is being called at 2025-06-29T01:18:00+0000 2025-06-29 03:17:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f885fe12]) (logid:12d3eb6e) Removed 0 old drs migration plans 2025-06-29 03:17:47,054 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-eae15733]) (logid:7d4584aa) Resetting hosts suitable for reconnect 2025-06-29 03:17:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-eae15733]) (logid:7d4584aa) Completed resetting hosts suitable for reconnect 2025-06-29 03:17:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-eae15733]) (logid:7d4584aa) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:17:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-eae15733]) (logid:7d4584aa) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:17:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-eae15733]) (logid:7d4584aa) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:17:47,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-eae15733]) (logid:7d4584aa) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:17:47,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:17:47,217 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:17:47,219 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:17:47,222 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:17:47,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c245595]) (logid:220b7bf1) No inactive management server node found 2025-06-29 03:17:47,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c245595]) (logid:220b7bf1) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:17:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d76ce3f2]) (logid:47bd0b18) No inactive management server node found 2025-06-29 03:17:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d76ce3f2]) (logid:47bd0b18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:50,154 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-19d409cf]) (logid:1cf9e037) HA health check task is running... 2025-06-29 03:17:50,158 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) StorageCollector is running... 2025-06-29 03:17:50,164 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:17:50,164 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:17:50,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:17:50,167 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 2-6717118844223095493: Routed from 90520738888109 2025-06-29 03:17:50,253 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 4-6717118844223095493: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:17:50,257 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:17:50,257 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:17:50,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:17:50,259 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 2-3048092522799696742: Routed from 90520738888109 2025-06-29 03:17:50,305 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 1-3048092522799696742: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:17:50,330 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:17:50,330 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:17:50,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:17:50,331 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 2-3048092522799696743: Routed from 90520738888109 2025-06-29 03:17:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67d36144]) (logid:8d7e16a5) No inactive management server node found 2025-06-29 03:17:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67d36144]) (logid:8d7e16a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:50,419 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 1-3048092522799696743: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:17:50,428 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:17:50,429 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:17:50,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:17:50,429 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 2-3048092522799696744: Routed from 90520738888109 2025-06-29 03:17:51,521 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 1-3048092522799696744: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:17:51,523 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:17:51,523 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:17:51,523 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:17:51,524 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 2-1612007191621930026: Routed from 90520738888109 2025-06-29 03:17:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab541510]) (logid:e22224d5) No inactive management server node found 2025-06-29 03:17:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab541510]) (logid:e22224d5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:52,586 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-de73fb84]) (logid:b4174fce) Seq 2-1612007191621930026: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:17:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321fbff0]) (logid:74fc1cfa) No inactive management server node found 2025-06-29 03:17:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321fbff0]) (logid:74fc1cfa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:54,155 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ff5b39c1]) (logid:f5571b48) HA health check task is running... 2025-06-29 03:17:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e792ab9b]) (logid:e7d5efec) No inactive management server node found 2025-06-29 03:17:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e792ab9b]) (logid:e7d5efec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:55,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8105: Seq 7-8105: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:17:55,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8105 2025-06-29 03:17:56,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e5f05f1]) (logid:6b06a1f8) No inactive management server node found 2025-06-29 03:17:56,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e5f05f1]) (logid:6b06a1f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0d1ab430]) (logid:52a74127) Enabled non-edge zones available for scan: 1 2025-06-29 03:17:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0d1ab430]) (logid:52a74127) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:17:56,765 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0d1ab430]) (logid:52a74127) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:17:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2584c8ab]) (logid:d88d8570) Enabled non-edge zones available for scan: 1 2025-06-29 03:17:56,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2584c8ab]) (logid:d88d8570) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:17:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef0f842e]) (logid:722cc272) No inactive management server node found 2025-06-29 03:17:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef0f842e]) (logid:722cc272) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:17:57,935 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-8acd33fe]) (logid:7a44a529) AutoScaling Monitor is running... 2025-06-29 03:17:58,061 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-7cf140d2]) (logid:361e1d2e) Starting to remove all volume_stats rows older than [Sat Jun 28 15:17:58 CEST 2025]. 2025-06-29 03:17:58,062 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-7cf140d2]) (logid:361e1d2e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:17:58 CEST 2025]. 2025-06-29 03:17:58,156 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-deb5936c]) (logid:f0456424) HA health check task is running... 2025-06-29 03:17:59,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-413f35c9]) (logid:862a1af7) No inactive management server node found 2025-06-29 03:17:59,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-413f35c9]) (logid:862a1af7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84f9b5fd]) (logid:13eabfe7) No inactive management server node found 2025-06-29 03:18:00,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84f9b5fd]) (logid:13eabfe7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:18:02,158 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-342f452b]) (logid:bb7f032d) HA health check task is running... 2025-06-29 03:18:02,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7074a232]) (logid:4c5ea872) No inactive management server node found 2025-06-29 03:18:02,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7074a232]) (logid:4c5ea872) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd8b7b24]) (logid:82ba88db) No inactive management server node found 2025-06-29 03:18:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd8b7b24]) (logid:82ba88db) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:04,655 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:18:04,658 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:18:04,658 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) Seq 2-3048092522799696745: Routed from 90520738888109 2025-06-29 03:18:04,671 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) Seq 1-3048092522799696745: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:18:04,722 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:18:04,723 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) Seq 2-1612007191621930027: Routed from 90520738888109 2025-06-29 03:18:04,768 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-fbee36ce]) (logid:4ee25254) Seq 2-1612007191621930027: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:18:05,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8106: Seq 7-8106: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:05,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8106 2025-06-29 03:18:05,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97e88055]) (logid:dba16120) No inactive management server node found 2025-06-29 03:18:05,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97e88055]) (logid:dba16120) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:06,159 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ab81e936]) (logid:1b99fee2) HA health check task is running... 2025-06-29 03:18:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15625b46]) (logid:5b92080d) No inactive management server node found 2025-06-29 03:18:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15625b46]) (logid:5b92080d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4027e335]) (logid:627fd1c0) No inactive management server node found 2025-06-29 03:18:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4027e335]) (logid:627fd1c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:09,826 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a0ce160]) (logid:8b833cfa) No inactive management server node found 2025-06-29 03:18:09,826 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a0ce160]) (logid:8b833cfa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:10,160 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-0405de1f]) (logid:952eed78) HA health check task is running... 2025-06-29 03:18:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-542abff5]) (logid:8916390f) Found 0 routers to update status. 2025-06-29 03:18:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-542abff5]) (logid:8916390f) Found 0 VPC's to update Redundant State. 2025-06-29 03:18:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-542abff5]) (logid:8916390f) Found 0 networks to update RvR status. 2025-06-29 03:18:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-970fdba7]) (logid:865a5cd7) No inactive management server node found 2025-06-29 03:18:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-970fdba7]) (logid:865a5cd7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:18:12,836 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-982aa535]) (logid:24eaa6b1) No inactive management server node found 2025-06-29 03:18:12,836 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-982aa535]) (logid:24eaa6b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:13,885 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-07b0cb3e]) (logid:b476bee1) Starting to remove all vm_stats rows older than [Sat Jun 28 15:18:13 CEST 2025]. 2025-06-29 03:18:13,923 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-07b0cb3e]) (logid:b476bee1) Removed a total of [6] vm_stats rows older than [Sat Jun 28 15:18:13 CEST 2025]. 2025-06-29 03:18:14,162 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f7ec95a0]) (logid:53d767a4) HA health check task is running... 2025-06-29 03:18:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac19bd14]) (logid:ef54e2b7) No inactive management server node found 2025-06-29 03:18:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac19bd14]) (logid:ef54e2b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:15,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8107: Seq 7-8107: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:15,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8107 2025-06-29 03:18:15,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8fe5f42]) (logid:6d380779) No inactive management server node found 2025-06-29 03:18:15,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8fe5f42]) (logid:6d380779) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7ed9fdd]) (logid:86235636) No inactive management server node found 2025-06-29 03:18:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7ed9fdd]) (logid:86235636) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:18,163 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-59d62f52]) (logid:4450c0a6) HA health check task is running... 2025-06-29 03:18:18,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c26a013]) (logid:13739d85) No inactive management server node found 2025-06-29 03:18:18,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c26a013]) (logid:13739d85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:20,355 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b7d86e5]) (logid:d02343a6) No inactive management server node found 2025-06-29 03:18:20,355 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b7d86e5]) (logid:d02343a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:21,836 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8b28185]) (logid:14ffcb6e) No inactive management server node found 2025-06-29 03:18:21,836 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8b28185]) (logid:14ffcb6e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:22,164 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1edcc4cb]) (logid:fec041e3) HA health check task is running... 2025-06-29 03:18:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68113c6f]) (logid:c365c412) No inactive management server node found 2025-06-29 03:18:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68113c6f]) (logid:c365c412) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:24,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d761df0]) (logid:4c775240) No inactive management server node found 2025-06-29 03:18:24,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d761df0]) (logid:4c775240) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:25,153 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8108: Seq 7-8108: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:25,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8108 2025-06-29 03:18:26,166 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3de303a4]) (logid:b1b17509) HA health check task is running... 2025-06-29 03:18:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a891fd0f]) (logid:b1313f4c) No inactive management server node found 2025-06-29 03:18:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a891fd0f]) (logid:b1313f4c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c9b6d87c]) (logid:75674f98) Enabled non-edge zones available for scan: 1 2025-06-29 03:18:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c9b6d87c]) (logid:75674f98) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:18:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c9b6d87c]) (logid:75674f98) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:18:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c74b94ab]) (logid:e6b3f845) Enabled non-edge zones available for scan: 1 2025-06-29 03:18:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c74b94ab]) (logid:e6b3f845) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:18:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8be339c5]) (logid:0e5913a4) No inactive management server node found 2025-06-29 03:18:27,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8be339c5]) (logid:0e5913a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9388545c]) (logid:0c704937) No inactive management server node found 2025-06-29 03:18:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9388545c]) (logid:0c704937) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:30,152 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8109: Seq 7-8109: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:30,167 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-fb3d07c1]) (logid:c92f64fb) HA health check task is running... 2025-06-29 03:18:30,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8109 2025-06-29 03:18:30,609 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:18:30,610 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:18:30,612 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) Seq 2-3048092522799696746: Routed from 90520738888109 2025-06-29 03:18:30,655 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) Seq 1-3048092522799696746: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:18:30,657 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:18:30,657 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) Seq 2-1612007191621930028: Routed from 90520738888109 2025-06-29 03:18:30,660 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-c53189e2]) (logid:9bb24982) Seq 2-1612007191621930028: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:18:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4975f4bf]) (logid:873e8cb7) No inactive management server node found 2025-06-29 03:18:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4975f4bf]) (logid:873e8cb7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:32,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbd11164]) (logid:3596a268) No inactive management server node found 2025-06-29 03:18:32,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbd11164]) (logid:3596a268) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:18:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec10c702]) (logid:2e9a5e30) No inactive management server node found 2025-06-29 03:18:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec10c702]) (logid:2e9a5e30) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:33,984 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:18:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:18:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:18:33,988 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:18:34,169 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0f2c0d33]) (logid:ea17fa24) HA health check task is running... 2025-06-29 03:18:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dce0c50]) (logid:0d19f6ee) No inactive management server node found 2025-06-29 03:18:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dce0c50]) (logid:0d19f6ee) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08bf2c0d]) (logid:40ae6b9a) No inactive management server node found 2025-06-29 03:18:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08bf2c0d]) (logid:40ae6b9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:38,170 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b88cd267]) (logid:e393bf3f) HA health check task is running... 2025-06-29 03:18:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3411001d]) (logid:60dfd223) No inactive management server node found 2025-06-29 03:18:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3411001d]) (logid:60dfd223) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:18:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6dffe44]) (logid:972fb8ad) No inactive management server node found 2025-06-29 03:18:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6dffe44]) (logid:972fb8ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:40,153 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8110: Seq 7-8110: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:40,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8110 2025-06-29 03:18:40,816 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-c0d2b007]) (logid:f296d945) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:18:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-c0d2b007]) (logid:f296d945) Snapshot scheduler is being called at [2025-06-29T01:18:40+0000]. 2025-06-29 03:18:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-c0d2b007]) (logid:f296d945) There are [0] scheduled snapshots to be executed at [2025-06-29T01:18:40+0000]. 2025-06-29 03:18:40,916 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-7788dd1b]) (logid:2f607b42) External devices are not used. Skipping external device usage collection 2025-06-29 03:18:40,946 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-24e39b68]) (logid:7b07cbac) checking health of usage server 2025-06-29 03:18:40,947 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-24e39b68]) (logid:7b07cbac) usage server running? false, heartbeat: null 2025-06-29 03:18:40,950 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-24e39b68]) (logid:7b07cbac) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 03:18:40,951 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-24e39b68]) (logid:7b07cbac) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 03:18:40,980 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-20a70903]) (logid:d44eed03) Backup backup.poll is being called at 2025-06-29T01:18:40+0000 2025-06-29 03:18:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-cd8183df]) (logid:30d81355) Found 0 running routers. 2025-06-29 03:18:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b71bc83d]) (logid:a839b494) Found 0 routers to update status. 2025-06-29 03:18:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b71bc83d]) (logid:a839b494) Found 0 VPC's to update Redundant State. 2025-06-29 03:18:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b71bc83d]) (logid:a839b494) Found 0 networks to update RvR status. 2025-06-29 03:18:41,018 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Found 1 running routers. 2025-06-29 03:18:41,022 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:18:41,023 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Seq 2-1612007191621930029: Routed from 90520738888109 2025-06-29 03:18:41,023 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Seq 2-1612007191621930029: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:41,050 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-d3e4dff5]) (logid:427c1c17) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 03:18:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-a9541445]) (logid:a6b868d7) DbCollector is running... 2025-06-29 03:18:41,285 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-f7375034]) (logid:95f2cfa8) Started resource counters recalculation periodic task. 2025-06-29 03:18:41,288 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-f7375034]) (logid:95f2cfa8) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:18:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faf923b6]) (logid:f2011175) No inactive management server node found 2025-06-29 03:18:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faf923b6]) (logid:f2011175) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:41,644 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-f7375034]) (logid:95f2cfa8) Finished resource counters recalculation periodic task. 2025-06-29 03:18:41,676 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-1612007191621930029: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:41,676 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Seq 2-1612007191621930029: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:18:41,676 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 03:18:41,679 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 03:18:41,716 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 03:18:41,717 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Seq 2-1612007191621930030: Routed from 90520738888109 2025-06-29 03:18:41,718 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Seq 2-1612007191621930030: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) ManagementServerCollector is running... 2025-06-29 03:18:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) DB properties were already loaded 2025-06-29 03:18:42,154 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:18:42,169 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Execution of process [4063428] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:18:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Exit value of process [4063428] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:18:42,169 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Process [4063428] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:18:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) usage install: null 2025-06-29 03:18:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Sessions active: 0 2025-06-29 03:18:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:18:42,171 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-007e95e3]) (logid:d2c1a03c) HA health check task is running... 2025-06-29 03:18:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063431] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:18:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:18:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063436] for command [/bin/bash -c uname -r ]. 2025-06-29 03:18:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:18:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063437] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:18:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) system memory from /proc: 4887515136 2025-06-29 03:18:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:18:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063442] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:18:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) free memory from /proc: 496164864 2025-06-29 03:18:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:18:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063447] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:18:42,208 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) used memory from /proc: 769184 2025-06-29 03:18:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:18:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063448] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:18:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:18:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063449] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:18:42,226 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:18:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063450] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:18:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:18:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063457] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:18:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:18:42,245 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Successfully executed process [4063458] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:18:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:18:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail buffersdirect.count, 120 2025-06-29 03:18:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail buffersdirect.used, 658750 2025-06-29 03:18:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail buffersmapped.count, 7 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail buffersmapped.used, 798602 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail gcPS-Scavenge.count, 512 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail gcPS-Scavenge.time, 3733 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail jvmuptime, 43862324 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memoryheap.committed, 357040128 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memoryheap.init, 77594624 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memoryheap.usage, 0.10033036011714458 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorynon-heap.max, -1 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorynon-heap.usage, -2.20652168E8 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorynon-heap.used, 220652168 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30298138767620886 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.Metaspace.usage, 0.9857870041270981 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.PS-Eden-Space.usage, 0.009475312883038604 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.PS-Old-Gen.usage, 0.12393348830735365 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorypools.PS-Survivor-Space.usage, 0.5564933189978967 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorytotal.committed, 582156288 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorytotal.init, 85262336 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorytotal.max, 1908932607 2025-06-29 03:18:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail memorytotal.used, 412230664 2025-06-29 03:18:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail threadsdeadlocks, [] 2025-06-29 03:18:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail threadsnew.count, 0 2025-06-29 03:18:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:18:42,255 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-5481f513]) (logid:219edba5) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:18:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166325248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43862247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":411367048,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191523896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496164864,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.17,0.09],"systemCyclesUsage":[66373430,53813083,4588243330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:18:42,256 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1461, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:18:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166325248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43862247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":411367048,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191523896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496164864,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.17,0.09],"systemCyclesUsage":[66373430,53813083,4588243330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:18:42,256 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:18:42,260 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-328:[ctx-9d72dc06]) (logid:8bd4457e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:18:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166325248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43862247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":411367048,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191523896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496164864,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.17,0.09],"systemCyclesUsage":[66373430,53813083,4588243330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:18:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:18:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1461, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:18:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357040128,"freeJvmMemoryBytes":166325248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43862247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":411367048,"totalCommitted":582156288,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":191523896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496164864,"systemMemoryUsed":769184,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.17,0.09],"systemCyclesUsage":[66373430,53813083,4588243330],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:18:42,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e54f280]) (logid:26cd910e) No inactive management server node found 2025-06-29 03:18:42,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e54f280]) (logid:26cd910e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:43,439 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 2-1612007191621930030: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.b5fd950b-a770-4570-a8d5-91a122720709","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:43,439 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-be518ec6]) (logid:7efa090f) Seq 2-1612007191621930030: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 03:18:43,440 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 03:18:43,444 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:18:43,444 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Seq 2-1612007191621930031: Routed from 90520738888109 2025-06-29 03:18:43,445 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Seq 2-1612007191621930031: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:43,880 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[]) (logid:) Seq 2-1612007191621930031: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "16.95704460144043", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "26.86142921447754", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "36.72480583190918", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "47.806501388549805", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159882489", "lastRunDuration": "47.652482986450195", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159882559", "lastRunDuration": "8092.227220535278", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159890693", "lastRunDuration": "94.23685073852539", "message": "Memory Usage within limits with current at 49.3719%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159890787", "lastRunDuration": "3245.7752227783203", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159894078", "lastRunDuration": "41.30673408508301", "message": "Sufficient free space is 2571.49609375 MB"}, "lastRun": {"start": "2025-06-29 01:18:02.305040", "end": "2025-06-29 01:18:14.119689", "duration": "11.814649820327759"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751159401792", "lastRunDuration": "38.95878791809082", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751159401831", "lastRunDuration": "50.12941360473633", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751159402005", "lastRunDuration": "52.835702896118164", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751159402167", "lastRunDuration": "41.292667388916016", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:10:01.790715", "end": "2025-06-29 01:10:02.251920", "duration": "0.46120524406433105"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "16.95704460144043", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "26.86142921447754", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "36.72480583190918", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "47.806501388549805", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159882489", "lastRunDuration": "47.652482986450195", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159882559", "lastRunDuration": "8092.227220535278", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159890693", "lastRunDuration": "94.23685073852539", "message": "Memory Usage within limits with current at 49.3719%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159890787", "lastRunDuration": "3245.7752227783203", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159894078", "lastRunDuration": "41.30673408508301", "message": "Sufficient free space is 2571.49609375 MB"}, "lastRun": {"start": "2025-06-29 01:18:02.305040", "end": "2025-06-29 01:18:14.119689", "duration": "11.814649820327759"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751159401792", "lastRunDuration": "38.95878791809082", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751159401831", "lastRunDuration": "50.12941360473633", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751159402005", "lastRunDuration": "52.835702896118164", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751159402167", "lastRunDuration": "41.292667388916016", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:10:01.790715", "end": "2025-06-29 01:10:02.251920", "duration": "0.46120524406433105"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:43,881 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Seq 2-1612007191621930031: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:18:43,881 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "16.95704460144043", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "26.86142921447754", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "36.72480583190918", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "47.806501388549805", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159882489", "lastRunDuration": "47.652482986450195", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159882559", "lastRunDuration": "8092.227220535278", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159890693", "lastRunDuration": "94.23685073852539", "message": "Memory Usage within limits with current at 49.3719%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159890787", "lastRunDuration": "3245.7752227783203", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159894078", "lastRunDuration": "41.30673408508301", "message": "Sufficient free space is 2571.49609375 MB"}, "lastRun": {"start": "2025-06-29 01:18:02.305040", "end": "2025-06-29 01:18:14.119689", "duration": "11.814649820327759"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751159401792", "lastRunDuration": "38.95878791809082", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751159401831", "lastRunDuration": "50.12941360473633", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751159402005", "lastRunDuration": "52.835702896118164", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751159402167", "lastRunDuration": "41.292667388916016", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:10:01.790715", "end": "2025-06-29 01:10:02.251920", "duration": "0.46120524406433105"}}} 2025-06-29 03:18:43,915 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "16.95704460144043", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "26.86142921447754", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "36.72480583190918", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751159882308", "lastRunDuration": "47.806501388549805", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751159882489", "lastRunDuration": "47.652482986450195", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751159882559", "lastRunDuration": "8092.227220535278", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751159890693", "lastRunDuration": "94.23685073852539", "message": "Memory Usage within limits with current at 49.3719%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751159890787", "lastRunDuration": "3245.7752227783203", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751159894078", "lastRunDuration": "41.30673408508301", "message": "Sufficient free space is 2571.49609375 MB"}, "lastRun": {"start": "2025-06-29 01:18:02.305040", "end": "2025-06-29 01:18:14.119689", "duration": "11.814649820327759"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751159401792", "lastRunDuration": "38.95878791809082", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751159401831", "lastRunDuration": "50.12941360473633", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751159402005", "lastRunDuration": "52.835702896118164", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751159402167", "lastRunDuration": "41.292667388916016", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:10:01.790715", "end": "2025-06-29 01:10:02.251920", "duration": "0.46120524406433105"}}} 2025-06-29 03:18:43,916 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found check types executed on VR basic, start: 2025-06-29 01:18:02.305040, end: 2025-06-29 01:18:14.119689, duration: 11.814649820327759 2025-06-29 03:18:43,926 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6a5bc13e- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 03:18:02 CEST 2025, details: service is running which took running duration (ms) 16.95704460144043 2025-06-29 03:18:43,937 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5add383e- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 03:18:02 CEST 2025, details: service is running which took running duration (ms) 26.86142921447754 2025-06-29 03:18:43,948 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@182b8413- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 03:18:02 CEST 2025, details: service is running which took running duration (ms) 36.72480583190918 2025-06-29 03:18:43,959 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@33e5c0ba- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 03:18:02 CEST 2025, details: service is running which took running duration (ms) 47.806501388549805 2025-06-29 03:18:43,969 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@703a5c9- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 03:18:02 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 47.652482986450195 2025-06-29 03:18:43,982 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@34ac61df- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 03:18:02 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8092.227220535278 2025-06-29 03:18:43,993 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@70e37045- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 03:18:10 CEST 2025, details: Memory Usage within limits with current at 49.3719% which took running duration (ms) 94.23685073852539 2025-06-29 03:18:44,009 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6d1e8ef3- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 03:18:10 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3245.7752227783203 2025-06-29 03:18:44,021 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@77a94374- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 03:18:14 CEST 2025, details: Sufficient free space is 2571.49609375 MB which took running duration (ms) 41.30673408508301 2025-06-29 03:18:44,021 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found check types executed on VR advanced, start: 2025-06-29 01:10:01.790715, end: 2025-06-29 01:10:02.251920, duration: 0.46120524406433105 2025-06-29 03:18:44,032 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@205c66c7- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 03:10:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 38.95878791809082 2025-06-29 03:18:44,043 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4b564bd1- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 03:10:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 50.12941360473633 2025-06-29 03:18:44,053 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@52a62d5e- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 03:10:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 52.835702896118164 2025-06-29 03:18:44,065 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-0146a453]) (logid:6458da94) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6fa9769d- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 03:10:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 41.292667388916016 2025-06-29 03:18:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c752d43]) (logid:0dcf16ab) No inactive management server node found 2025-06-29 03:18:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c752d43]) (logid:0dcf16ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:18:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:18:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a632849]) (logid:e381650c) No inactive management server node found 2025-06-29 03:18:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a632849]) (logid:e381650c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d61f6f42]) (logid:7e50dca7) VM scheduler.poll is being called at 2025-06-29T01:19:00+0000 2025-06-29 03:18:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d61f6f42]) (logid:7e50dca7) Got 0 scheduled jobs to be executed at 2025-06-29T01:19:00+0000 2025-06-29 03:18:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d61f6f42]) (logid:7e50dca7) Cleaned up 0 VM scheduled job entries 2025-06-29 03:18:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-6871e936]) (logid:67c0373f) VM Operation Thread Running 2025-06-29 03:18:46,173 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5228e91d]) (logid:3936090a) HA health check task is running... 2025-06-29 03:18:46,188 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-3:[ctx-ec028c28]) (logid:eb7a8382) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:18:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9f5f7916]) (logid:35b94f47) ClusterDRS.poll is being called at 2025-06-29T01:19:00+0000 2025-06-29 03:18:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9f5f7916]) (logid:35b94f47) Removed 0 old drs migration plans 2025-06-29 03:18:47,208 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:18:47,210 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:18:47,212 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:18:47,215 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:18:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97dedb07]) (logid:d913c215) No inactive management server node found 2025-06-29 03:18:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97dedb07]) (logid:d913c215) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:48,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62fc147b]) (logid:b75c5378) No inactive management server node found 2025-06-29 03:18:48,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62fc147b]) (logid:b75c5378) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:50,153 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8112: Seq 7-8112: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:18:50,174 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-bb157d1e]) (logid:bea92d4a) HA health check task is running... 2025-06-29 03:18:50,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8112 2025-06-29 03:18:50,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6ddcb20]) (logid:77b93933) No inactive management server node found 2025-06-29 03:18:50,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6ddcb20]) (logid:77b93933) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:51,029 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-c3805e5b]) (logid:239ec5e7) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:18:51,037 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-c3805e5b]) (logid:239ec5e7) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:18:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6744c636]) (logid:4add5f6f) No inactive management server node found 2025-06-29 03:18:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6744c636]) (logid:4add5f6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:52,586 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) StorageCollector is running... 2025-06-29 03:18:52,591 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:18:52,591 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:18:52,591 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:18:52,592 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-6717118844223095494: Routed from 90520738888109 2025-06-29 03:18:52,671 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 4-6717118844223095494: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:18:52,675 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:18:52,675 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:18:52,675 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:18:52,676 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-1612007191621930032: Routed from 90520738888109 2025-06-29 03:18:52,727 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-1612007191621930032: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:18:52,749 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:18:52,749 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:18:52,749 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:18:52,750 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-3048092522799696747: Routed from 90520738888109 2025-06-29 03:18:52,797 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 1-3048092522799696747: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:18:52,809 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:18:52,809 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:18:52,809 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:18:52,810 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-3048092522799696748: Routed from 90520738888109 2025-06-29 03:18:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98e54cd2]) (logid:187ddf15) No inactive management server node found 2025-06-29 03:18:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98e54cd2]) (logid:187ddf15) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:53,902 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 1-3048092522799696748: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:18:53,904 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:18:53,904 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:18:53,904 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:18:53,905 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-1612007191621930033: Routed from 90520738888109 2025-06-29 03:18:54,175 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1c29686a]) (logid:3852a24e) HA health check task is running... 2025-06-29 03:18:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcec0e44]) (logid:44f3ada7) No inactive management server node found 2025-06-29 03:18:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcec0e44]) (logid:44f3ada7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:54,966 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-42dede35]) (logid:9007fabe) Seq 2-1612007191621930033: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:18:56,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:18:56,210 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Seq 2-3048092522799696749: Routed from 90520738888109 2025-06-29 03:18:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b065ffe7]) (logid:5429bbed) No inactive management server node found 2025-06-29 03:18:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b065ffe7]) (logid:5429bbed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-66e22946]) (logid:5607aa89) Enabled non-edge zones available for scan: 1 2025-06-29 03:18:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-66e22946]) (logid:5607aa89) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:18:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-66e22946]) (logid:5607aa89) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:18:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-478dc556]) (logid:8fbae12f) Enabled non-edge zones available for scan: 1 2025-06-29 03:18:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-478dc556]) (logid:8fbae12f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:18:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63b69c36]) (logid:582fca09) No inactive management server node found 2025-06-29 03:18:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63b69c36]) (logid:582fca09) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:18:57,937 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-cc1e4d26]) (logid:7cc2d6a8) AutoScaling Monitor is running... 2025-06-29 03:18:58,064 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-152f2277]) (logid:d14b259f) Starting to remove all volume_stats rows older than [Sat Jun 28 15:18:58 CEST 2025]. 2025-06-29 03:18:58,064 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-152f2277]) (logid:d14b259f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:18:58 CEST 2025]. 2025-06-29 03:18:58,176 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-078c27be]) (logid:e1137d11) HA health check task is running... 2025-06-29 03:18:59,237 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Seq 1-3048092522799696749: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:18:59,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 03:18:59,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:18:59,266 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Seq 2-1612007191621930034: Routed from 90520738888109 2025-06-29 03:18:59,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bedabb2f]) (logid:de9c5be1) No inactive management server node found 2025-06-29 03:18:59,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bedabb2f]) (logid:de9c5be1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:00,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8113: Seq 7-8113: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:00,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8113 2025-06-29 03:19:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4e13fc2]) (logid:0887acde) No inactive management server node found 2025-06-29 03:19:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4e13fc2]) (logid:0887acde) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:02,177 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f726b1b7]) (logid:9dba41e9) HA health check task is running... 2025-06-29 03:19:02,342 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b6ba911]) (logid:657c67b4) No inactive management server node found 2025-06-29 03:19:02,342 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b6ba911]) (logid:657c67b4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-160fcd5a]) (logid:67bd4fc2) No inactive management server node found 2025-06-29 03:19:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-160fcd5a]) (logid:67bd4fc2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:04,377 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Seq 2-1612007191621930034: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:19:04,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f788d177]) (logid:d96c3692) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 03:19:04,821 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:19:04,825 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:19:04,826 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) Seq 2-3048092522799696750: Routed from 90520738888109 2025-06-29 03:19:04,838 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) Seq 1-3048092522799696750: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:19:04,872 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:19:04,873 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) Seq 2-1612007191621930035: Routed from 90520738888109 2025-06-29 03:19:04,921 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-52f6933a]) (logid:d1c10f5c) Seq 2-1612007191621930035: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:19:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0d00238]) (logid:e307f065) No inactive management server node found 2025-06-29 03:19:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0d00238]) (logid:e307f065) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:06,179 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3a52c072]) (logid:8d1cb0e4) HA health check task is running... 2025-06-29 03:19:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9d989dd]) (logid:04b87087) No inactive management server node found 2025-06-29 03:19:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9d989dd]) (logid:04b87087) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:08,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39306ff2]) (logid:d38004d8) No inactive management server node found 2025-06-29 03:19:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39306ff2]) (logid:d38004d8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e6bdfb5]) (logid:b0228d4a) No inactive management server node found 2025-06-29 03:19:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e6bdfb5]) (logid:b0228d4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:10,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8114: Seq 7-8114: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:10,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8114 2025-06-29 03:19:10,180 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-dd948dd2]) (logid:08b96cb6) HA health check task is running... 2025-06-29 03:19:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-80457652]) (logid:8324a60b) Found 0 routers to update status. 2025-06-29 03:19:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-80457652]) (logid:8324a60b) Found 0 VPC's to update Redundant State. 2025-06-29 03:19:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-80457652]) (logid:8324a60b) Found 0 networks to update RvR status. 2025-06-29 03:19:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Running Capacity Checker ... 2025-06-29 03:19:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) recalculating system capacity 2025-06-29 03:19:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Executing cpu/ram capacity update 2025-06-29 03:19:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:11,079 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:11,090 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:19:11,090 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:19:11,090 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:19:11,100 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:11,102 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:11,106 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:11,107 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:19:11,107 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:19:11,107 DEBUG [c.c.c.CapacityManagerImpl] (pool-320-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:19:11,116 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Done executing cpu/ram capacity update 2025-06-29 03:19:11,117 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Executing storage capacity update 2025-06-29 03:19:11,123 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:19:11,123 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:19:11,123 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:19:11,125 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:19:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:19:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:19:11,130 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:19:11,132 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:19:11,137 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:19:11,137 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:19:11,137 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:19:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:19:11,145 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:19:11,145 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:19:11,145 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:19:11,149 DEBUG [c.c.s.StorageManagerImpl] (pool-321-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:19:11,149 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Done executing storage capacity update 2025-06-29 03:19:11,149 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Executing capacity updates for public ip and Vlans 2025-06-29 03:19:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Done capacity updates for public ip and Vlans 2025-06-29 03:19:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Executing capacity updates for private ip 2025-06-29 03:19:11,173 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Done executing capacity updates for private ip 2025-06-29 03:19:11,173 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Done recalculating system capacity 2025-06-29 03:19:11,201 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c22dd18a]) (logid:050ea657) Done running Capacity Checker ... 2025-06-29 03:19:11,260 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-dca422fa]) (logid:8e148e61) Checking if any host reservation can be released ... 2025-06-29 03:19:11,264 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-dca422fa]) (logid:8e148e61) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:11,267 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-dca422fa]) (logid:8e148e61) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:11,267 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-dca422fa]) (logid:8e148e61) Done running HostReservationReleaseChecker ... 2025-06-29 03:19:11,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6722e56]) (logid:1b065d7c) No inactive management server node found 2025-06-29 03:19:11,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6722e56]) (logid:1b065d7c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:12,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47ac4a28]) (logid:7fd9f82d) No inactive management server node found 2025-06-29 03:19:12,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47ac4a28]) (logid:7fd9f82d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:13,924 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-0434a8ee]) (logid:45b12012) Starting to remove all vm_stats rows older than [Sat Jun 28 15:19:13 CEST 2025]. 2025-06-29 03:19:13,948 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-0434a8ee]) (logid:45b12012) Removed a total of [7] vm_stats rows older than [Sat Jun 28 15:19:13 CEST 2025]. 2025-06-29 03:19:14,181 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dea90467]) (logid:3d00568d) HA health check task is running... 2025-06-29 03:19:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d66881a]) (logid:b0308015) No inactive management server node found 2025-06-29 03:19:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d66881a]) (logid:b0308015) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:15,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8115: Seq 7-8115: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:15,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8115 2025-06-29 03:19:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0c79225]) (logid:0e23fa4a) No inactive management server node found 2025-06-29 03:19:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0c79225]) (logid:0e23fa4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-af8824a3]) (logid:246f2287) Resetting hosts suitable for reconnect 2025-06-29 03:19:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-af8824a3]) (logid:246f2287) Completed resetting hosts suitable for reconnect 2025-06-29 03:19:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-af8824a3]) (logid:246f2287) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:19:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-af8824a3]) (logid:246f2287) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:19:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-af8824a3]) (logid:246f2287) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:19:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-af8824a3]) (logid:246f2287) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:19:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3042f8cc]) (logid:79ad0825) No inactive management server node found 2025-06-29 03:19:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3042f8cc]) (logid:79ad0825) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:19:18,183 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b77ae8d7]) (logid:b47e2ea0) HA health check task is running... 2025-06-29 03:19:18,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c95197b]) (logid:aa603da6) No inactive management server node found 2025-06-29 03:19:18,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c95197b]) (logid:aa603da6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf0875d7]) (logid:cc674463) No inactive management server node found 2025-06-29 03:19:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf0875d7]) (logid:cc674463) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:21,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b5b6060]) (logid:9e928e4f) No inactive management server node found 2025-06-29 03:19:21,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b5b6060]) (logid:9e928e4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:22,184 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-35433858]) (logid:60453af1) HA health check task is running... 2025-06-29 03:19:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc422744]) (logid:0f40b460) No inactive management server node found 2025-06-29 03:19:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc422744]) (logid:0f40b460) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f2504dd]) (logid:7be475d2) No inactive management server node found 2025-06-29 03:19:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f2504dd]) (logid:7be475d2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:25,155 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8116: Seq 7-8116: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:25,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8116 2025-06-29 03:19:26,186 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4269586f]) (logid:8d1606ce) HA health check task is running... 2025-06-29 03:19:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4f41cb0]) (logid:e212900c) No inactive management server node found 2025-06-29 03:19:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4f41cb0]) (logid:e212900c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e68d7c91]) (logid:42f6e6ba) Enabled non-edge zones available for scan: 1 2025-06-29 03:19:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e68d7c91]) (logid:42f6e6ba) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:19:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e68d7c91]) (logid:42f6e6ba) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:19:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-eb2d8d26]) (logid:210ed628) Enabled non-edge zones available for scan: 1 2025-06-29 03:19:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-eb2d8d26]) (logid:210ed628) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:19:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a7fce96]) (logid:030fa855) No inactive management server node found 2025-06-29 03:19:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a7fce96]) (logid:030fa855) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d865979d]) (logid:cd6cc53f) No inactive management server node found 2025-06-29 03:19:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d865979d]) (logid:cd6cc53f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:30,187 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a26d87e4]) (logid:779e9ef5) HA health check task is running... 2025-06-29 03:19:30,662 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:19:30,663 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:19:30,664 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) Seq 2-3048092522799696751: Routed from 90520738888109 2025-06-29 03:19:30,707 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) Seq 1-3048092522799696751: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:19:30,708 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:19:30,709 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) Seq 2-1612007191621930036: Routed from 90520738888109 2025-06-29 03:19:30,712 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ee33dcc1]) (logid:8d522c9e) Seq 2-1612007191621930036: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:19:30,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d98cd30]) (logid:a46a22ea) No inactive management server node found 2025-06-29 03:19:30,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d98cd30]) (logid:a46a22ea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce5cbc66]) (logid:41dbde52) No inactive management server node found 2025-06-29 03:19:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce5cbc66]) (logid:41dbde52) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0978a54e]) (logid:9e992fe6) No inactive management server node found 2025-06-29 03:19:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0978a54e]) (logid:9e992fe6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:19:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:19:34,188 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-79be2aee]) (logid:bbd11fdc) HA health check task is running... 2025-06-29 03:19:35,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8117: Seq 7-8117: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:35,177 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8117 2025-06-29 03:19:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc830ff8]) (logid:c9c291fe) No inactive management server node found 2025-06-29 03:19:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc830ff8]) (logid:c9c291fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:36,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc660ae7]) (logid:5bf8834c) No inactive management server node found 2025-06-29 03:19:36,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc660ae7]) (logid:5bf8834c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:38,189 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-703a5459]) (logid:9338adf8) HA health check task is running... 2025-06-29 03:19:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad01af45]) (logid:c5c9a875) No inactive management server node found 2025-06-29 03:19:38,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad01af45]) (logid:c5c9a875) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99c580b0]) (logid:0286f624) No inactive management server node found 2025-06-29 03:19:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99c580b0]) (logid:0286f624) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-12b0809d]) (logid:a285ae2d) Found 0 routers to update status. 2025-06-29 03:19:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-12b0809d]) (logid:a285ae2d) Found 0 VPC's to update Redundant State. 2025-06-29 03:19:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-12b0809d]) (logid:a285ae2d) Found 0 networks to update RvR status. 2025-06-29 03:19:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-51dff80c]) (logid:d995244c) DbCollector is running... 2025-06-29 03:19:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c323821]) (logid:80125e6f) No inactive management server node found 2025-06-29 03:19:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c323821]) (logid:80125e6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) ManagementServerCollector is running... 2025-06-29 03:19:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) DB properties were already loaded 2025-06-29 03:19:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:19:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Execution of process [4063467] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:19:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Exit value of process [4063467] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:19:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Process [4063467] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:19:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) usage install: null 2025-06-29 03:19:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Sessions active: 0 2025-06-29 03:19:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:19:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063470] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:19:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:19:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063475] for command [/bin/bash -c uname -r ]. 2025-06-29 03:19:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:19:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063476] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:19:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) system memory from /proc: 4887515136 2025-06-29 03:19:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:19:42,190 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-60bf1171]) (logid:448e6942) HA health check task is running... 2025-06-29 03:19:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063481] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:19:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) free memory from /proc: 488681472 2025-06-29 03:19:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:19:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063486] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:19:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) used memory from /proc: 769312 2025-06-29 03:19:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:19:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063487] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:19:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:19:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063488] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:19:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:19:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063489] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:19:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:19:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063496] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:19:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:19:42,246 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Successfully executed process [4063497] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:19:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:19:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail buffersdirect.count, 120 2025-06-29 03:19:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail buffersdirect.used, 658750 2025-06-29 03:19:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:19:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail buffersmapped.count, 7 2025-06-29 03:19:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail buffersmapped.used, 798602 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail gcPS-Scavenge.count, 513 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail gcPS-Scavenge.time, 3739 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail jvmuptime, 43922325 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memoryheap.committed, 357564416 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memoryheap.init, 77594624 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memoryheap.usage, 0.10566783507948752 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorynon-heap.max, -1 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.PS-Eden-Space.usage, 0.023968466346897612 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.PS-Old-Gen.usage, 0.12476308369365903 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorypools.PS-Survivor-Space.usage, 0.4855810312124399 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorytotal.committed, 582680576 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorytotal.init, 85262336 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorytotal.max, 1908932607 2025-06-29 03:19:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail memorytotal.used, 422492952 2025-06-29 03:19:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail threadsdeadlocks, [] 2025-06-29 03:19:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail threadsnew.count, 0 2025-06-29 03:19:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:19:42,251 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-37e5e525]) (logid:da19556c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:19:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":156651056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43922247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":421632672,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201727992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488681472,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66373766,53813411,4588266421],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:19:42,252 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1463, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:19:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":156651056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43922247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":421632672,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201727992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488681472,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66373766,53813411,4588266421],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:19:42,252 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:19:42,256 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-328:[ctx-4f0962e2]) (logid:cb09f4a6) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:19:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":156651056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43922247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":421632672,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201727992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488681472,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66373766,53813411,4588266421],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:19:42,256 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:19:42,256 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1463, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:19:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":156651056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43922247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":421632672,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":201727992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":488681472,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66373766,53813411,4588266421],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:19:42,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c8ca9c7]) (logid:0aea140e) No inactive management server node found 2025-06-29 03:19:42,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c8ca9c7]) (logid:0aea140e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:44,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c11cc937]) (logid:4028564e) No inactive management server node found 2025-06-29 03:19:44,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c11cc937]) (logid:4028564e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:45,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8118: Seq 7-8118: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:45,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8118 2025-06-29 03:19:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:19:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:19:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b146d01b]) (logid:dd611d36) No inactive management server node found 2025-06-29 03:19:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b146d01b]) (logid:dd611d36) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7db65f4c]) (logid:febaf66a) VM scheduler.poll is being called at 2025-06-29T01:20:00+0000 2025-06-29 03:19:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7db65f4c]) (logid:febaf66a) Got 0 scheduled jobs to be executed at 2025-06-29T01:20:00+0000 2025-06-29 03:19:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7db65f4c]) (logid:febaf66a) Cleaned up 0 VM scheduled job entries 2025-06-29 03:19:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-41a11061]) (logid:0c23edd6) VM Operation Thread Running 2025-06-29 03:19:46,191 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-80478530]) (logid:52079a8e) HA health check task is running... 2025-06-29 03:19:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-0c775bf9]) (logid:52e01bd3) ClusterDRS.poll is being called at 2025-06-29T01:20:00+0000 2025-06-29 03:19:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-0c775bf9]) (logid:52e01bd3) Removed 0 old drs migration plans 2025-06-29 03:19:47,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:47,229 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:47,230 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:19:47,233 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:19:47,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4030b74]) (logid:aaeb94c7) No inactive management server node found 2025-06-29 03:19:47,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4030b74]) (logid:aaeb94c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5e94b44]) (logid:0f0a5779) No inactive management server node found 2025-06-29 03:19:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5e94b44]) (logid:0f0a5779) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:50,193 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-025912be]) (logid:aa5b14ad) HA health check task is running... 2025-06-29 03:19:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-850f82d3]) (logid:af99deb1) No inactive management server node found 2025-06-29 03:19:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-850f82d3]) (logid:af99deb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc4b70f3]) (logid:4f3b47be) No inactive management server node found 2025-06-29 03:19:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc4b70f3]) (logid:4f3b47be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:53,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd5902f3]) (logid:d960ad32) No inactive management server node found 2025-06-29 03:19:53,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd5902f3]) (logid:d960ad32) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:54,194 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d5e00b5e]) (logid:ef80f827) HA health check task is running... 2025-06-29 03:19:54,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0595bd0]) (logid:126fb30f) No inactive management server node found 2025-06-29 03:19:54,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0595bd0]) (logid:126fb30f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:54,966 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) StorageCollector is running... 2025-06-29 03:19:54,972 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:19:54,972 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:19:54,972 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:19:54,974 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 2-6717118844223095495: Routed from 90520738888109 2025-06-29 03:19:55,031 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 4-6717118844223095495: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:19:55,035 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:19:55,035 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:19:55,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:19:55,037 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 2-3048092522799696752: Routed from 90520738888109 2025-06-29 03:19:55,084 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 1-3048092522799696752: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:19:55,106 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:19:55,107 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:19:55,107 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:19:55,107 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 2-3048092522799696753: Routed from 90520738888109 2025-06-29 03:19:55,157 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8120: Seq 7-8120: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:19:55,192 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8120 2025-06-29 03:19:55,196 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 1-3048092522799696753: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:19:55,207 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:19:55,207 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:19:55,207 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:19:55,208 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 2-3048092522799696754: Routed from 90520738888109 2025-06-29 03:19:56,299 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 1-3048092522799696754: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:19:56,301 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:19:56,301 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:19:56,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:19:56,302 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 2-1612007191621930037: Routed from 90520738888109 2025-06-29 03:19:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcd97d46]) (logid:e6402598) No inactive management server node found 2025-06-29 03:19:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcd97d46]) (logid:e6402598) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d01e3271]) (logid:0a4e1158) Enabled non-edge zones available for scan: 1 2025-06-29 03:19:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d01e3271]) (logid:0a4e1158) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:19:56,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d01e3271]) (logid:0a4e1158) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:19:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-98296bcc]) (logid:ce4d74f3) Enabled non-edge zones available for scan: 1 2025-06-29 03:19:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-98296bcc]) (logid:ce4d74f3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:19:57,363 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b6fd765a]) (logid:82200125) Seq 2-1612007191621930037: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:19:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb010e9a]) (logid:d8fbecf0) No inactive management server node found 2025-06-29 03:19:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb010e9a]) (logid:d8fbecf0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:19:57,940 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-16cc09c3]) (logid:bafc4cb8) AutoScaling Monitor is running... 2025-06-29 03:19:58,066 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-76e00cd3]) (logid:ef78b02d) Starting to remove all volume_stats rows older than [Sat Jun 28 15:19:58 CEST 2025]. 2025-06-29 03:19:58,067 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-76e00cd3]) (logid:ef78b02d) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:19:58 CEST 2025]. 2025-06-29 03:19:58,195 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e3961fe4]) (logid:de1a8de1) HA health check task is running... 2025-06-29 03:19:59,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-270e5837]) (logid:32fee934) No inactive management server node found 2025-06-29 03:19:59,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-270e5837]) (logid:32fee934) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad500537]) (logid:67ec6a4d) No inactive management server node found 2025-06-29 03:20:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad500537]) (logid:67ec6a4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:02,196 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-97fd7912]) (logid:75d74880) HA health check task is running... 2025-06-29 03:20:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f8a6fb5]) (logid:0c7448cf) No inactive management server node found 2025-06-29 03:20:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f8a6fb5]) (logid:0c7448cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:03,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-823e2f45]) (logid:16f51a7f) No inactive management server node found 2025-06-29 03:20:03,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-823e2f45]) (logid:16f51a7f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:20:04,975 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:20:04,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:20:04,979 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) Seq 2-3048092522799696755: Routed from 90520738888109 2025-06-29 03:20:04,992 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) Seq 1-3048092522799696755: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:20:05,031 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:20:05,032 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) Seq 2-1612007191621930038: Routed from 90520738888109 2025-06-29 03:20:05,095 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-55a44e07]) (logid:4498e373) Seq 2-1612007191621930038: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:20:05,157 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8121: Seq 7-8121: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:20:05,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8121 2025-06-29 03:20:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c45857e]) (logid:4d6d4ec6) No inactive management server node found 2025-06-29 03:20:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c45857e]) (logid:4d6d4ec6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:06,198 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-aa76af1a]) (logid:e3a4bacd) HA health check task is running... 2025-06-29 03:20:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee066159]) (logid:446016bd) No inactive management server node found 2025-06-29 03:20:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee066159]) (logid:446016bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-780e437f]) (logid:88277714) No inactive management server node found 2025-06-29 03:20:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-780e437f]) (logid:88277714) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e458b1d8]) (logid:7d1245c6) No inactive management server node found 2025-06-29 03:20:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e458b1d8]) (logid:7d1245c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:10,199 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-49aad272]) (logid:e759008a) HA health check task is running... 2025-06-29 03:20:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0daddb74]) (logid:b4891f92) Found 0 routers to update status. 2025-06-29 03:20:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0daddb74]) (logid:b4891f92) Found 0 VPC's to update Redundant State. 2025-06-29 03:20:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0daddb74]) (logid:b4891f92) Found 0 networks to update RvR status. 2025-06-29 03:20:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f11fb893]) (logid:fa6a8d2b) No inactive management server node found 2025-06-29 03:20:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f11fb893]) (logid:fa6a8d2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74939b98]) (logid:371369b3) No inactive management server node found 2025-06-29 03:20:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74939b98]) (logid:371369b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:13,950 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-98d85eaa]) (logid:b1702420) Starting to remove all vm_stats rows older than [Sat Jun 28 15:20:13 CEST 2025]. 2025-06-29 03:20:13,983 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-98d85eaa]) (logid:b1702420) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:20:13 CEST 2025]. 2025-06-29 03:20:14,200 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-303d93c8]) (logid:9d906dd1) HA health check task is running... 2025-06-29 03:20:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9334a75]) (logid:66f724eb) No inactive management server node found 2025-06-29 03:20:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9334a75]) (logid:66f724eb) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:20:14,800 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:20:15,157 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8122: Seq 7-8122: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:20:15,178 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8122 2025-06-29 03:20:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfec66f3]) (logid:ae640d38) No inactive management server node found 2025-06-29 03:20:15,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfec66f3]) (logid:ae640d38) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc563bc0]) (logid:9f9801ff) No inactive management server node found 2025-06-29 03:20:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc563bc0]) (logid:9f9801ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:20:18,202 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-eea3320b]) (logid:b3f3e4f1) HA health check task is running... 2025-06-29 03:20:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d204cbbd]) (logid:fdcb37a7) No inactive management server node found 2025-06-29 03:20:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d204cbbd]) (logid:fdcb37a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:20,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8123: Seq 7-8123: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:20:20,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8123 2025-06-29 03:20:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63794974]) (logid:42124d16) No inactive management server node found 2025-06-29 03:20:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63794974]) (logid:42124d16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2586dc7c]) (logid:86df48f4) No inactive management server node found 2025-06-29 03:20:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2586dc7c]) (logid:86df48f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:22,204 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2d01ed64]) (logid:7c0a4f2c) HA health check task is running... 2025-06-29 03:20:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-118f89f7]) (logid:21298e85) No inactive management server node found 2025-06-29 03:20:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-118f89f7]) (logid:21298e85) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:24,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a37b888e]) (logid:60fd5af8) No inactive management server node found 2025-06-29 03:20:24,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a37b888e]) (logid:60fd5af8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:26,205 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d02bd8aa]) (logid:a84eeaf4) HA health check task is running... 2025-06-29 03:20:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1792059b]) (logid:f3d92796) No inactive management server node found 2025-06-29 03:20:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1792059b]) (logid:f3d92796) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d011b309]) (logid:0143bf58) Enabled non-edge zones available for scan: 1 2025-06-29 03:20:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d011b309]) (logid:0143bf58) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:20:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d011b309]) (logid:0143bf58) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:20:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3a48c668]) (logid:27b0299f) Enabled non-edge zones available for scan: 1 2025-06-29 03:20:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3a48c668]) (logid:27b0299f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:20:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a51f320]) (logid:ef70d2cf) No inactive management server node found 2025-06-29 03:20:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a51f320]) (logid:ef70d2cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:29,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-238194cd]) (logid:818c467b) No inactive management server node found 2025-06-29 03:20:29,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-238194cd]) (logid:818c467b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:30,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8124: Seq 7-8124: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:20:30,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8124 2025-06-29 03:20:30,206 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f17c1e04]) (logid:20fdd1c1) HA health check task is running... 2025-06-29 03:20:30,714 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:20:30,715 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:20:30,717 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) Seq 2-3048092522799696756: Routed from 90520738888109 2025-06-29 03:20:30,759 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) Seq 1-3048092522799696756: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:20:30,760 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:20:30,761 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) Seq 2-1612007191621930039: Routed from 90520738888109 2025-06-29 03:20:30,764 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-909df434]) (logid:adda2704) Seq 2-1612007191621930039: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:20:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e085333]) (logid:415caad1) No inactive management server node found 2025-06-29 03:20:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e085333]) (logid:415caad1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ab8f239]) (logid:c80ca47c) No inactive management server node found 2025-06-29 03:20:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ab8f239]) (logid:c80ca47c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfc9376b]) (logid:d2e24bff) No inactive management server node found 2025-06-29 03:20:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfc9376b]) (logid:d2e24bff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:20:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:20:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:20:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:20:34,207 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c14b7132]) (logid:67906411) HA health check task is running... 2025-06-29 03:20:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf762ba5]) (logid:e4373e1c) No inactive management server node found 2025-06-29 03:20:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf762ba5]) (logid:e4373e1c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e22468f]) (logid:6d752fcb) No inactive management server node found 2025-06-29 03:20:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e22468f]) (logid:6d752fcb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:38,208 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-745225b9]) (logid:4f2c3fae) HA health check task is running... 2025-06-29 03:20:38,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-318039ba]) (logid:298070ac) No inactive management server node found 2025-06-29 03:20:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-318039ba]) (logid:298070ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1a78f92]) (logid:f2a01fc0) No inactive management server node found 2025-06-29 03:20:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1a78f92]) (logid:f2a01fc0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:40,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8125: Seq 7-8125: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:20:40,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8125 2025-06-29 03:20:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9fba75c0]) (logid:90b39f69) Found 0 routers to update status. 2025-06-29 03:20:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9fba75c0]) (logid:90b39f69) Found 0 VPC's to update Redundant State. 2025-06-29 03:20:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9fba75c0]) (logid:90b39f69) Found 0 networks to update RvR status. 2025-06-29 03:20:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-01646dd9]) (logid:1cf00afc) DbCollector is running... 2025-06-29 03:20:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-491df310]) (logid:16971bd1) No inactive management server node found 2025-06-29 03:20:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-491df310]) (logid:16971bd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) ManagementServerCollector is running... 2025-06-29 03:20:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) DB properties were already loaded 2025-06-29 03:20:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:20:42,168 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Execution of process [4063512] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:20:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Exit value of process [4063512] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:20:42,168 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Process [4063512] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:20:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) usage install: null 2025-06-29 03:20:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Sessions active: 0 2025-06-29 03:20:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:20:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063515] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:20:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:20:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063520] for command [/bin/bash -c uname -r ]. 2025-06-29 03:20:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:20:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063521] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:20:42,186 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) system memory from /proc: 4887515136 2025-06-29 03:20:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:20:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063526] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:20:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) free memory from /proc: 490455040 2025-06-29 03:20:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:20:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063531] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:20:42,208 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) used memory from /proc: 769312 2025-06-29 03:20:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:20:42,210 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-42c0e80b]) (logid:b9faefbc) HA health check task is running... 2025-06-29 03:20:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063532] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:20:42,214 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:20:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063533] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:20:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:20:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063534] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:20:42,239 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:20:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063541] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:20:42,245 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:20:42,251 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Successfully executed process [4063542] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:20:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:20:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail buffersdirect.count, 120 2025-06-29 03:20:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail buffersdirect.used, 658750 2025-06-29 03:20:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:20:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail buffersmapped.count, 7 2025-06-29 03:20:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail buffersmapped.used, 798602 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail gcPS-Scavenge.count, 514 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail gcPS-Scavenge.time, 3744 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail jvmuptime, 43982331 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memoryheap.committed, 357564416 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memoryheap.init, 77594624 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memoryheap.usage, 0.10207489105869996 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorynon-heap.max, -1 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.PS-Eden-Space.usage, 0.014053158785492431 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.PS-Old-Gen.usage, 0.12564010368690784 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorypools.PS-Survivor-Space.usage, 0.39063204251802885 2025-06-29 03:20:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorytotal.committed, 582680576 2025-06-29 03:20:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorytotal.init, 85262336 2025-06-29 03:20:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorytotal.max, 1908932607 2025-06-29 03:20:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail memorytotal.used, 415640408 2025-06-29 03:20:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail threadsdeadlocks, [] 2025-06-29 03:20:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail threadsnew.count, 0 2025-06-29 03:20:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:20:42,260 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-57034a18]) (logid:caeb2869) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:20:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":163518664,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43982246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":414778456,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":194867488,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490455040,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66374109,53813748,4588289513],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:20:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1465, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:20:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":163518664,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43982246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":414778456,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":194867488,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490455040,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66374109,53813748,4588289513],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:20:42,260 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:20:42,265 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:20:42,265 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-329:[ctx-a3ebb85a]) (logid:61b7b570) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:20:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":163518664,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43982246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":414778456,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":194867488,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490455040,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66374109,53813748,4588289513],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:20:42,265 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1465, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:20:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":357564416,"freeJvmMemoryBytes":163518664,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":43982246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":414778456,"totalCommitted":582680576,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":194867488,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490455040,"systemMemoryUsed":769312,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.15,0.09],"systemCyclesUsage":[66374109,53813748,4588289513],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:20:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bfa4926]) (logid:8692dd25) No inactive management server node found 2025-06-29 03:20:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bfa4926]) (logid:8692dd25) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-352f0ad6]) (logid:e6dd25ee) No inactive management server node found 2025-06-29 03:20:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-352f0ad6]) (logid:e6dd25ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:20:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:20:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-598e4775]) (logid:df019c49) No inactive management server node found 2025-06-29 03:20:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-598e4775]) (logid:df019c49) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d44e178c]) (logid:7613fa11) VM scheduler.poll is being called at 2025-06-29T01:21:00+0000 2025-06-29 03:20:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d44e178c]) (logid:7613fa11) Got 0 scheduled jobs to be executed at 2025-06-29T01:21:00+0000 2025-06-29 03:20:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d44e178c]) (logid:7613fa11) Cleaned up 0 VM scheduled job entries 2025-06-29 03:20:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-235de80e]) (logid:9d3fb31c) VM Operation Thread Running 2025-06-29 03:20:46,211 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-735afb46]) (logid:92a68d86) HA health check task is running... 2025-06-29 03:20:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6a0f1b04]) (logid:6eb437f9) ClusterDRS.poll is being called at 2025-06-29T01:21:00+0000 2025-06-29 03:20:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6a0f1b04]) (logid:6eb437f9) Removed 0 old drs migration plans 2025-06-29 03:20:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-21b60b45]) (logid:18dd94eb) Resetting hosts suitable for reconnect 2025-06-29 03:20:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-21b60b45]) (logid:18dd94eb) Completed resetting hosts suitable for reconnect 2025-06-29 03:20:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-21b60b45]) (logid:18dd94eb) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:20:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-21b60b45]) (logid:18dd94eb) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:20:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-21b60b45]) (logid:18dd94eb) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:20:47,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-21b60b45]) (logid:18dd94eb) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:20:47,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:20:47,246 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:20:47,247 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:20:47,250 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:20:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fad73514]) (logid:dbadf3d7) No inactive management server node found 2025-06-29 03:20:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fad73514]) (logid:dbadf3d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e4fae5c]) (logid:a4b14001) No inactive management server node found 2025-06-29 03:20:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e4fae5c]) (logid:a4b14001) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:50,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8127: Seq 7-8127: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:20:50,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8127 2025-06-29 03:20:50,212 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bac46e8f]) (logid:911bc3d8) HA health check task is running... 2025-06-29 03:20:50,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52eec634]) (logid:b31b2946) No inactive management server node found 2025-06-29 03:20:50,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52eec634]) (logid:b31b2946) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:20:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c413aa8]) (logid:2833736f) No inactive management server node found 2025-06-29 03:20:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c413aa8]) (logid:2833736f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ca57c1a]) (logid:00c855df) No inactive management server node found 2025-06-29 03:20:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ca57c1a]) (logid:00c855df) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:54,214 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bf411cc9]) (logid:10bf5743) HA health check task is running... 2025-06-29 03:20:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6236e675]) (logid:b9fd642d) No inactive management server node found 2025-06-29 03:20:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6236e675]) (logid:b9fd642d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c26243ca]) (logid:8955e252) No inactive management server node found 2025-06-29 03:20:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c26243ca]) (logid:8955e252) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-31b53991]) (logid:fba3747b) Enabled non-edge zones available for scan: 1 2025-06-29 03:20:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-31b53991]) (logid:fba3747b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:20:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-31b53991]) (logid:fba3747b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:20:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f600abed]) (logid:fc07481e) Enabled non-edge zones available for scan: 1 2025-06-29 03:20:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f600abed]) (logid:fc07481e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:20:57,364 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) StorageCollector is running... 2025-06-29 03:20:57,370 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:20:57,370 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:20:57,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:20:57,371 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-6717118844223095496: Routed from 90520738888109 2025-06-29 03:20:57,430 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 4-6717118844223095496: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:20:57,434 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:20:57,434 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:20:57,434 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:20:57,435 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-1612007191621930040: Routed from 90520738888109 2025-06-29 03:20:57,504 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-1612007191621930040: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:20:57,526 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:20:57,526 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:20:57,526 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:20:57,527 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-1612007191621930041: Routed from 90520738888109 2025-06-29 03:20:57,621 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-1612007191621930041: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:20:57,632 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:20:57,632 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:20:57,632 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:20:57,633 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-3048092522799696757: Routed from 90520738888109 2025-06-29 03:20:57,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddd3640b]) (logid:40ec6023) No inactive management server node found 2025-06-29 03:20:57,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddd3640b]) (logid:40ec6023) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:57,942 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-6ccd5a9f]) (logid:6e6f7e7f) AutoScaling Monitor is running... 2025-06-29 03:20:58,069 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-8d1cfc38]) (logid:9d7ea270) Starting to remove all volume_stats rows older than [Sat Jun 28 15:20:58 CEST 2025]. 2025-06-29 03:20:58,070 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-8d1cfc38]) (logid:9d7ea270) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:20:58 CEST 2025]. 2025-06-29 03:20:58,215 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6db54d01]) (logid:37d52366) HA health check task is running... 2025-06-29 03:20:58,684 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 1-3048092522799696757: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:20:58,686 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:20:58,686 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:20:58,686 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:20:58,687 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-1612007191621930042: Routed from 90520738888109 2025-06-29 03:20:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5221130]) (logid:b62a3bf0) No inactive management server node found 2025-06-29 03:20:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5221130]) (logid:b62a3bf0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:20:59,750 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-99d44691]) (logid:09270e67) Seq 2-1612007191621930042: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:21:00,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8128: Seq 7-8128: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:00,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8128 2025-06-29 03:21:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92b2cbf5]) (logid:3bd0ca1e) No inactive management server node found 2025-06-29 03:21:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92b2cbf5]) (logid:3bd0ca1e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:02,216 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-87938b50]) (logid:3131fbf4) HA health check task is running... 2025-06-29 03:21:02,343 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0f29db3]) (logid:0ade7694) No inactive management server node found 2025-06-29 03:21:02,343 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0f29db3]) (logid:0ade7694) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5fe2679]) (logid:a81ba6b9) No inactive management server node found 2025-06-29 03:21:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5fe2679]) (logid:a81ba6b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:05,151 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:21:05,155 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:21:05,156 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) Seq 2-3048092522799696758: Routed from 90520738888109 2025-06-29 03:21:05,168 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) Seq 1-3048092522799696758: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:21:05,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8129: Seq 7-8129: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:05,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8129 2025-06-29 03:21:05,202 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:21:05,203 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) Seq 2-1612007191621930043: Routed from 90520738888109 2025-06-29 03:21:05,320 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-9532d613]) (logid:08b41e8b) Seq 2-1612007191621930043: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:21:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8e2ebae]) (logid:32dec506) No inactive management server node found 2025-06-29 03:21:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8e2ebae]) (logid:32dec506) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:06,217 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4080a503]) (logid:bdb5ed5c) HA health check task is running... 2025-06-29 03:21:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b53d982c]) (logid:9e1eb2f3) No inactive management server node found 2025-06-29 03:21:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b53d982c]) (logid:9e1eb2f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f220dd2]) (logid:f9b12132) No inactive management server node found 2025-06-29 03:21:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f220dd2]) (logid:f9b12132) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acd15ca7]) (logid:550c70c8) No inactive management server node found 2025-06-29 03:21:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acd15ca7]) (logid:550c70c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:10,219 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e1c58f8c]) (logid:2720867b) HA health check task is running... 2025-06-29 03:21:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-49ef55c1]) (logid:a965a7a8) Found 0 routers to update status. 2025-06-29 03:21:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-49ef55c1]) (logid:a965a7a8) Found 0 VPC's to update Redundant State. 2025-06-29 03:21:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-49ef55c1]) (logid:a965a7a8) Found 0 networks to update RvR status. 2025-06-29 03:21:11,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a4ba69d]) (logid:9bfd84d9) No inactive management server node found 2025-06-29 03:21:11,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a4ba69d]) (logid:9bfd84d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0968a9ae]) (logid:023511e5) No inactive management server node found 2025-06-29 03:21:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0968a9ae]) (logid:023511e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:13,984 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-79505f9f]) (logid:b897f44d) Starting to remove all vm_stats rows older than [Sat Jun 28 15:21:13 CEST 2025]. 2025-06-29 03:21:14,020 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-79505f9f]) (logid:b897f44d) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:21:13 CEST 2025]. 2025-06-29 03:21:14,220 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ddfafebc]) (logid:211bddd6) HA health check task is running... 2025-06-29 03:21:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62ad81f5]) (logid:b717e34b) No inactive management server node found 2025-06-29 03:21:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62ad81f5]) (logid:b717e34b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:15,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8130: Seq 7-8130: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:15,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8130 2025-06-29 03:21:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47004450]) (logid:159f2755) No inactive management server node found 2025-06-29 03:21:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47004450]) (logid:159f2755) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b18fe5a]) (logid:d0420eee) No inactive management server node found 2025-06-29 03:21:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b18fe5a]) (logid:d0420eee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:18,221 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8258699b]) (logid:58197154) HA health check task is running... 2025-06-29 03:21:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7efda8cc]) (logid:0535770d) No inactive management server node found 2025-06-29 03:21:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7efda8cc]) (logid:0535770d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76d52161]) (logid:03ed1e88) No inactive management server node found 2025-06-29 03:21:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76d52161]) (logid:03ed1e88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:21,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbbedc96]) (logid:89f3b472) No inactive management server node found 2025-06-29 03:21:21,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbbedc96]) (logid:89f3b472) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:22,223 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-59acf90d]) (logid:1ed06a34) HA health check task is running... 2025-06-29 03:21:23,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0806cbf6]) (logid:a9496dae) No inactive management server node found 2025-06-29 03:21:23,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0806cbf6]) (logid:a9496dae) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e9afd0]) (logid:bb4728a8) No inactive management server node found 2025-06-29 03:21:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e9afd0]) (logid:bb4728a8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:25,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8131: Seq 7-8131: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:25,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8131 2025-06-29 03:21:26,224 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-dc7df2c9]) (logid:2f1e6c88) HA health check task is running... 2025-06-29 03:21:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cba2fe0]) (logid:4560cf18) No inactive management server node found 2025-06-29 03:21:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cba2fe0]) (logid:4560cf18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-78c0937f]) (logid:552a2127) Enabled non-edge zones available for scan: 1 2025-06-29 03:21:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-78c0937f]) (logid:552a2127) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:21:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-78c0937f]) (logid:552a2127) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:21:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-605a86b9]) (logid:da0e5aa5) Enabled non-edge zones available for scan: 1 2025-06-29 03:21:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-605a86b9]) (logid:da0e5aa5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:21:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-596aaa6f]) (logid:d91abbea) No inactive management server node found 2025-06-29 03:21:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-596aaa6f]) (logid:d91abbea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64184de0]) (logid:4e5ac621) No inactive management server node found 2025-06-29 03:21:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64184de0]) (logid:4e5ac621) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:30,226 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6425276a]) (logid:42a09fcd) HA health check task is running... 2025-06-29 03:21:30,766 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:21:30,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:21:30,768 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) Seq 2-3048092522799696759: Routed from 90520738888109 2025-06-29 03:21:30,811 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) Seq 1-3048092522799696759: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:21:30,812 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:21:30,813 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) Seq 2-1612007191621930044: Routed from 90520738888109 2025-06-29 03:21:30,816 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-6560ab88]) (logid:ef716815) Seq 2-1612007191621930044: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:21:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6113a6da]) (logid:b7199c87) No inactive management server node found 2025-06-29 03:21:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6113a6da]) (logid:b7199c87) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f60815dc]) (logid:e0fb3ddf) No inactive management server node found 2025-06-29 03:21:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f60815dc]) (logid:e0fb3ddf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d501c50]) (logid:b1f7c677) No inactive management server node found 2025-06-29 03:21:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d501c50]) (logid:b1f7c677) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:33,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:21:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:21:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:21:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:21:34,227 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3f4b5e85]) (logid:bcf92edd) HA health check task is running... 2025-06-29 03:21:35,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8132: Seq 7-8132: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:35,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8132 2025-06-29 03:21:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa6767f9]) (logid:72dc9420) No inactive management server node found 2025-06-29 03:21:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa6767f9]) (logid:72dc9420) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:36,838 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73255cd4]) (logid:735f3849) No inactive management server node found 2025-06-29 03:21:36,838 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73255cd4]) (logid:735f3849) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 5ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:21:38,229 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cfc0a9d1]) (logid:8e3ed48a) HA health check task is running... 2025-06-29 03:21:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-316b7a34]) (logid:cfca7580) No inactive management server node found 2025-06-29 03:21:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-316b7a34]) (logid:cfca7580) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd6184a2]) (logid:2cf6a591) No inactive management server node found 2025-06-29 03:21:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd6184a2]) (logid:2cf6a591) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-70262cf2]) (logid:aa80460d) Found 0 routers to update status. 2025-06-29 03:21:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-70262cf2]) (logid:aa80460d) Found 0 VPC's to update Redundant State. 2025-06-29 03:21:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-70262cf2]) (logid:aa80460d) Found 0 networks to update RvR status. 2025-06-29 03:21:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-5e48b6b4]) (logid:bf904c5b) DbCollector is running... 2025-06-29 03:21:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb6bc35a]) (logid:52053fed) No inactive management server node found 2025-06-29 03:21:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb6bc35a]) (logid:52053fed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:21:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) ManagementServerCollector is running... 2025-06-29 03:21:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) DB properties were already loaded 2025-06-29 03:21:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:21:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Execution of process [4063548] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:21:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Exit value of process [4063548] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:21:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Process [4063548] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:21:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) usage install: null 2025-06-29 03:21:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Sessions active: 0 2025-06-29 03:21:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:21:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063551] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:21:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:21:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063556] for command [/bin/bash -c uname -r ]. 2025-06-29 03:21:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:21:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063557] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:21:42,186 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) system memory from /proc: 4887515136 2025-06-29 03:21:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:21:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063562] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:21:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) free memory from /proc: 487186432 2025-06-29 03:21:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:21:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063567] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:21:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) used memory from /proc: 769160 2025-06-29 03:21:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:21:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063568] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:21:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:21:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063569] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:21:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:21:42,230 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-86daaf6e]) (logid:b0fa66a5) HA health check task is running... 2025-06-29 03:21:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063570] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:21:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:21:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063577] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:21:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:21:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Successfully executed process [4063578] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail buffersdirect.count, 120 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail buffersdirect.used, 658750 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail buffersmapped.count, 7 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail buffersmapped.used, 798602 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail gcPS-Scavenge.count, 515 2025-06-29 03:21:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail gcPS-Scavenge.time, 3749 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail jvmuptime, 44042322 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memoryheap.committed, 358088704 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memoryheap.init, 77594624 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memoryheap.usage, 0.09810291217991494 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorynon-heap.max, -1 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.PS-Eden-Space.usage, 0.0031318403071151466 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.PS-Old-Gen.usage, 0.125708759856809 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorypools.PS-Survivor-Space.usage, 0.3786128117487981 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorytotal.committed, 583204864 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorytotal.init, 85262336 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorytotal.max, 1908932607 2025-06-29 03:21:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail memorytotal.used, 408051480 2025-06-29 03:21:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail threadsdeadlocks, [] 2025-06-29 03:21:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail threadsnew.count, 0 2025-06-29 03:21:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:21:42,249 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-85c927e4]) (logid:50c29b71) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:21:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":171654456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44042246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":407157856,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187271848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487186432,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.12,0.09],"systemCyclesUsage":[66374440,53814084,4588312619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:21:42,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1467, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:21:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":171654456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44042246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":407157856,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187271848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487186432,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.12,0.09],"systemCyclesUsage":[66374440,53814084,4588312619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:21:42,249 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:21:42,253 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:21:42,253 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-329:[ctx-b4f5e9a6]) (logid:cb681a15) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:21:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":171654456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44042246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":407157856,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187271848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487186432,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.12,0.09],"systemCyclesUsage":[66374440,53814084,4588312619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:21:42,253 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1467, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:21:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":171654456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44042246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":407157856,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":187271848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487186432,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.12,0.09],"systemCyclesUsage":[66374440,53814084,4588312619],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:21:42,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3da5e7f]) (logid:49dc922f) No inactive management server node found 2025-06-29 03:21:42,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3da5e7f]) (logid:49dc922f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8912b355]) (logid:27667251) No inactive management server node found 2025-06-29 03:21:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8912b355]) (logid:27667251) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:45,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8133: Seq 7-8133: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:45,182 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8133 2025-06-29 03:21:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:21:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:21:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b7df973]) (logid:af5318ba) No inactive management server node found 2025-06-29 03:21:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b7df973]) (logid:af5318ba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d2e98cc5]) (logid:8f6145da) VM scheduler.poll is being called at 2025-06-29T01:22:00+0000 2025-06-29 03:21:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d2e98cc5]) (logid:8f6145da) Got 0 scheduled jobs to be executed at 2025-06-29T01:22:00+0000 2025-06-29 03:21:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d2e98cc5]) (logid:8f6145da) Cleaned up 0 VM scheduled job entries 2025-06-29 03:21:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-8053859b]) (logid:c780742b) VM Operation Thread Running 2025-06-29 03:21:46,231 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b31df79f]) (logid:16544633) HA health check task is running... 2025-06-29 03:21:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d576aace]) (logid:7ed6cc4e) ClusterDRS.poll is being called at 2025-06-29T01:22:00+0000 2025-06-29 03:21:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d576aace]) (logid:7ed6cc4e) Removed 0 old drs migration plans 2025-06-29 03:21:47,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:21:47,258 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:21:47,259 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:21:47,262 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:21:47,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e230c5]) (logid:3676e42a) No inactive management server node found 2025-06-29 03:21:47,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e230c5]) (logid:3676e42a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9bdfe8]) (logid:b5c3980f) No inactive management server node found 2025-06-29 03:21:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9bdfe8]) (logid:b5c3980f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:50,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8135: Seq 7-8135: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:21:50,187 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8135 2025-06-29 03:21:50,232 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f500fca8]) (logid:705c7000) HA health check task is running... 2025-06-29 03:21:50,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-298e74ce]) (logid:0cf9c75e) No inactive management server node found 2025-06-29 03:21:50,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-298e74ce]) (logid:0cf9c75e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28f612a6]) (logid:2171bb82) No inactive management server node found 2025-06-29 03:21:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28f612a6]) (logid:2171bb82) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77855ac2]) (logid:46fe4f0b) No inactive management server node found 2025-06-29 03:21:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77855ac2]) (logid:46fe4f0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:54,234 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8958c349]) (logid:8a888f02) HA health check task is running... 2025-06-29 03:21:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66bc77e0]) (logid:a5b910e9) No inactive management server node found 2025-06-29 03:21:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66bc77e0]) (logid:a5b910e9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58794da2]) (logid:f460e0d0) No inactive management server node found 2025-06-29 03:21:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58794da2]) (logid:f460e0d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c4ec93e]) (logid:74c32a76) Enabled non-edge zones available for scan: 1 2025-06-29 03:21:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1c4ec93e]) (logid:74c32a76) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:21:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c4ec93e]) (logid:74c32a76) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:21:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e8512707]) (logid:84732e03) Enabled non-edge zones available for scan: 1 2025-06-29 03:21:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e8512707]) (logid:84732e03) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:21:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7355f03]) (logid:e8c33451) No inactive management server node found 2025-06-29 03:21:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7355f03]) (logid:e8c33451) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:57,944 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-6afad80c]) (logid:d8328294) AutoScaling Monitor is running... 2025-06-29 03:21:58,071 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-3717f4da]) (logid:9ebb152e) Starting to remove all volume_stats rows older than [Sat Jun 28 15:21:58 CEST 2025]. 2025-06-29 03:21:58,072 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-3717f4da]) (logid:9ebb152e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:21:58 CEST 2025]. 2025-06-29 03:21:58,236 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9d896b48]) (logid:5eba85ae) HA health check task is running... 2025-06-29 03:21:59,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b78a046]) (logid:b9d6ec5d) No inactive management server node found 2025-06-29 03:21:59,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b78a046]) (logid:b9d6ec5d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:21:59,750 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) StorageCollector is running... 2025-06-29 03:21:59,756 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:21:59,756 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:21:59,756 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:21:59,758 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-6717118844223095497: Routed from 90520738888109 2025-06-29 03:21:59,812 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 4-6717118844223095497: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:21:59,816 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:21:59,816 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:21:59,816 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:21:59,817 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-1612007191621930045: Routed from 90520738888109 2025-06-29 03:21:59,868 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-1612007191621930045: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:21:59,890 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:21:59,890 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:21:59,890 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:21:59,891 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-1612007191621930046: Routed from 90520738888109 2025-06-29 03:21:59,985 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-1612007191621930046: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:21:59,997 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:21:59,997 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:21:59,997 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:21:59,998 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-3048092522799696760: Routed from 90520738888109 2025-06-29 03:22:00,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8136: Seq 7-8136: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:00,172 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8136 2025-06-29 03:22:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f700234e]) (logid:6e49456f) No inactive management server node found 2025-06-29 03:22:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f700234e]) (logid:6e49456f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:01,079 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 1-3048092522799696760: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:22:01,081 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:22:01,081 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:22:01,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:22:01,082 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-1612007191621930047: Routed from 90520738888109 2025-06-29 03:22:02,143 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f527022e]) (logid:c063a4ce) Seq 2-1612007191621930047: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:22:02,237 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-50de5c31]) (logid:72193fe2) HA health check task is running... 2025-06-29 03:22:02,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d82418e]) (logid:1047a5ba) No inactive management server node found 2025-06-29 03:22:02,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d82418e]) (logid:1047a5ba) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-966d6c04]) (logid:c10a2d7d) No inactive management server node found 2025-06-29 03:22:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-966d6c04]) (logid:c10a2d7d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:05,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2966b87e]) (logid:c46f08b9) No inactive management server node found 2025-06-29 03:22:05,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2966b87e]) (logid:c46f08b9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:22:05,385 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:22:05,388 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:22:05,389 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) Seq 2-3048092522799696761: Routed from 90520738888109 2025-06-29 03:22:05,402 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) Seq 1-3048092522799696761: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:22:05,435 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:22:05,435 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) Seq 2-1612007191621930048: Routed from 90520738888109 2025-06-29 03:22:05,491 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-50081285]) (logid:5eb18578) Seq 2-1612007191621930048: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:22:06,238 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-17290451]) (logid:a3bbea8c) HA health check task is running... 2025-06-29 03:22:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-683a3470]) (logid:fbca2a0b) No inactive management server node found 2025-06-29 03:22:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-683a3470]) (logid:fbca2a0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-156168a2]) (logid:49bc46a4) No inactive management server node found 2025-06-29 03:22:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-156168a2]) (logid:49bc46a4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a999e3a]) (logid:70459a8a) No inactive management server node found 2025-06-29 03:22:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a999e3a]) (logid:70459a8a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:10,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8137: Seq 7-8137: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:10,171 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8137 2025-06-29 03:22:10,240 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-78c554b6]) (logid:61a0767a) HA health check task is running... 2025-06-29 03:22:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9a5262d]) (logid:a32819d2) Found 0 routers to update status. 2025-06-29 03:22:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9a5262d]) (logid:a32819d2) Found 0 VPC's to update Redundant State. 2025-06-29 03:22:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9a5262d]) (logid:a32819d2) Found 0 networks to update RvR status. 2025-06-29 03:22:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bb86618]) (logid:9ff17871) No inactive management server node found 2025-06-29 03:22:11,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bb86618]) (logid:9ff17871) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5aa50b16]) (logid:56260eab) No inactive management server node found 2025-06-29 03:22:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5aa50b16]) (logid:56260eab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:14,022 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-d2f587ae]) (logid:6ce16c87) Starting to remove all vm_stats rows older than [Sat Jun 28 15:22:14 CEST 2025]. 2025-06-29 03:22:14,049 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-d2f587ae]) (logid:6ce16c87) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:22:14 CEST 2025]. 2025-06-29 03:22:14,241 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7e7947c7]) (logid:66cd8788) HA health check task is running... 2025-06-29 03:22:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ec3079c]) (logid:1fc9e4bf) No inactive management server node found 2025-06-29 03:22:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ec3079c]) (logid:1fc9e4bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46bf235c]) (logid:bdebb54e) No inactive management server node found 2025-06-29 03:22:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46bf235c]) (logid:bdebb54e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:17,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c79f62e]) (logid:61de9813) Resetting hosts suitable for reconnect 2025-06-29 03:22:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c79f62e]) (logid:61de9813) Completed resetting hosts suitable for reconnect 2025-06-29 03:22:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c79f62e]) (logid:61de9813) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:22:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c79f62e]) (logid:61de9813) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:22:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c79f62e]) (logid:61de9813) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:22:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0c79f62e]) (logid:61de9813) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:22:17,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8efa3b3c]) (logid:6e4994b3) No inactive management server node found 2025-06-29 03:22:17,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8efa3b3c]) (logid:6e4994b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:18,243 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2788ddcf]) (logid:38276abe) HA health check task is running... 2025-06-29 03:22:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1778347]) (logid:1c5e84b2) No inactive management server node found 2025-06-29 03:22:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1778347]) (logid:1c5e84b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:20,164 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8138: Seq 7-8138: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:20,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8138 2025-06-29 03:22:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5a02892]) (logid:102cc28a) No inactive management server node found 2025-06-29 03:22:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5a02892]) (logid:102cc28a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0c165c6]) (logid:8bf321e0) No inactive management server node found 2025-06-29 03:22:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0c165c6]) (logid:8bf321e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:22,244 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-de8809b1]) (logid:ab3857b7) HA health check task is running... 2025-06-29 03:22:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20907ea7]) (logid:dab1b2d8) No inactive management server node found 2025-06-29 03:22:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20907ea7]) (logid:dab1b2d8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0932403d]) (logid:c256c325) No inactive management server node found 2025-06-29 03:22:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0932403d]) (logid:c256c325) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:26,245 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-82369a89]) (logid:bad6bf81) HA health check task is running... 2025-06-29 03:22:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23bfaaf3]) (logid:e507eab6) No inactive management server node found 2025-06-29 03:22:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23bfaaf3]) (logid:e507eab6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c49e503]) (logid:37ed11a6) Enabled non-edge zones available for scan: 1 2025-06-29 03:22:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0c49e503]) (logid:37ed11a6) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:22:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c49e503]) (logid:37ed11a6) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:22:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-26f71c8f]) (logid:ecb9c598) Enabled non-edge zones available for scan: 1 2025-06-29 03:22:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-26f71c8f]) (logid:ecb9c598) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:22:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ef7b824]) (logid:442572d0) No inactive management server node found 2025-06-29 03:22:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ef7b824]) (logid:442572d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d71fefd]) (logid:e09e3d36) No inactive management server node found 2025-06-29 03:22:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d71fefd]) (logid:e09e3d36) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:30,164 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8139: Seq 7-8139: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:30,185 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8139 2025-06-29 03:22:30,246 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-62f0cf65]) (logid:67216e7b) HA health check task is running... 2025-06-29 03:22:30,818 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:22:30,820 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:22:30,821 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) Seq 2-3048092522799696762: Routed from 90520738888109 2025-06-29 03:22:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04431e14]) (logid:cc2b0478) No inactive management server node found 2025-06-29 03:22:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04431e14]) (logid:cc2b0478) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:30,864 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) Seq 1-3048092522799696762: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:22:30,865 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:22:30,866 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) Seq 2-1612007191621930049: Routed from 90520738888109 2025-06-29 03:22:30,869 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-743b1944]) (logid:8d4870a8) Seq 2-1612007191621930049: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:22:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-952b7fd0]) (logid:4067689b) No inactive management server node found 2025-06-29 03:22:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-952b7fd0]) (logid:4067689b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efa754df]) (logid:8de61886) No inactive management server node found 2025-06-29 03:22:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efa754df]) (logid:8de61886) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:22:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:22:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:22:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:22:34,248 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-caa84280]) (logid:108e2c2c) HA health check task is running... 2025-06-29 03:22:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ad8df13]) (logid:7829c7f7) No inactive management server node found 2025-06-29 03:22:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ad8df13]) (logid:7829c7f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99379ac1]) (logid:22d84334) No inactive management server node found 2025-06-29 03:22:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99379ac1]) (logid:22d84334) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:38,249 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d71c4126]) (logid:bed302e6) HA health check task is running... 2025-06-29 03:22:38,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa2c0447]) (logid:aad87f94) No inactive management server node found 2025-06-29 03:22:38,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa2c0447]) (logid:aad87f94) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:22:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f8b9196]) (logid:e728d2d7) No inactive management server node found 2025-06-29 03:22:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f8b9196]) (logid:e728d2d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:40,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8140: Seq 7-8140: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:40,185 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8140 2025-06-29 03:22:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-355c191b]) (logid:1e6d84aa) Found 0 routers to update status. 2025-06-29 03:22:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-355c191b]) (logid:1e6d84aa) Found 0 VPC's to update Redundant State. 2025-06-29 03:22:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-355c191b]) (logid:1e6d84aa) Found 0 networks to update RvR status. 2025-06-29 03:22:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-9e68063c]) (logid:0a38bf8e) DbCollector is running... 2025-06-29 03:22:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2947ef2]) (logid:f004b4a5) No inactive management server node found 2025-06-29 03:22:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2947ef2]) (logid:f004b4a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) ManagementServerCollector is running... 2025-06-29 03:22:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) DB properties were already loaded 2025-06-29 03:22:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:22:42,167 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Execution of process [4063592] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:22:42,167 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Exit value of process [4063592] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:22:42,167 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Process [4063592] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:22:42,167 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) usage install: null 2025-06-29 03:22:42,167 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Sessions active: 0 2025-06-29 03:22:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:22:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063595] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:22:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:22:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063600] for command [/bin/bash -c uname -r ]. 2025-06-29 03:22:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:22:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063601] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:22:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) system memory from /proc: 4887515136 2025-06-29 03:22:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:22:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063606] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:22:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) free memory from /proc: 490676224 2025-06-29 03:22:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:22:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063611] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:22:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) used memory from /proc: 769160 2025-06-29 03:22:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:22:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063612] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:22:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:22:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063613] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:22:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:22:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063614] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:22:42,231 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:22:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063621] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:22:42,235 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:22:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Successfully executed process [4063622] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail buffersdirect.count, 120 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail buffersdirect.used, 658750 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail buffersmapped.count, 7 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail buffersmapped.used, 798602 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail gcPS-Scavenge.count, 515 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail gcPS-Scavenge.time, 3749 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail jvmuptime, 44102320 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memoryheap.committed, 358088704 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memoryheap.init, 77594624 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memoryheap.usage, 0.11079641005325631 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorynon-heap.max, -1 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:22:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.PS-Eden-Space.usage, 0.038315109868859484 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.PS-Old-Gen.usage, 0.125708759856809 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorypools.PS-Survivor-Space.usage, 0.3786128117487981 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorytotal.committed, 583204864 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorytotal.init, 85262336 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorytotal.max, 1908932607 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail memorytotal.used, 432277312 2025-06-29 03:22:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail threadsdeadlocks, [] 2025-06-29 03:22:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail threadsnew.count, 0 2025-06-29 03:22:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:22:42,247 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-7f197aca]) (logid:610d3e3b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:22:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":147327712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44102244,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.21,"totalInit":85262336,"totalUsed":431480304,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":211502880,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490676224,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.21,0.14,0.1],"systemCyclesUsage":[66374770,53814400,4588335735],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:22:42,248 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1469, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:22:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":147327712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44102244,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.21,"totalInit":85262336,"totalUsed":431480304,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":211502880,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490676224,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.21,0.14,0.1],"systemCyclesUsage":[66374770,53814400,4588335735],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:22:42,248 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:22:42,250 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e72474d6]) (logid:126d343b) HA health check task is running... 2025-06-29 03:22:42,252 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-329:[ctx-6fc868dd]) (logid:2dc630b2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:22:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":147327712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44102244,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.21,"totalInit":85262336,"totalUsed":431480304,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":211502880,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490676224,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.21,0.14,0.1],"systemCyclesUsage":[66374770,53814400,4588335735],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:22:42,252 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:22:42,252 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1469, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:22:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":147327712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44102244,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.21,"totalInit":85262336,"totalUsed":431480304,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":211502880,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":490676224,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.21,0.14,0.1],"systemCyclesUsage":[66374770,53814400,4588335735],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:22:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d58f5c64]) (logid:11220ba2) No inactive management server node found 2025-06-29 03:22:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d58f5c64]) (logid:11220ba2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c49e30b4]) (logid:5117c570) No inactive management server node found 2025-06-29 03:22:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c49e30b4]) (logid:5117c570) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:22:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:22:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60b3eadf]) (logid:dbb98454) No inactive management server node found 2025-06-29 03:22:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60b3eadf]) (logid:dbb98454) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e9819b74]) (logid:0e276152) VM scheduler.poll is being called at 2025-06-29T01:23:00+0000 2025-06-29 03:22:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e9819b74]) (logid:0e276152) Got 0 scheduled jobs to be executed at 2025-06-29T01:23:00+0000 2025-06-29 03:22:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e9819b74]) (logid:0e276152) Cleaned up 0 VM scheduled job entries 2025-06-29 03:22:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-1e1bb5ae]) (logid:5994fa47) VM Operation Thread Running 2025-06-29 03:22:46,251 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a8518a9e]) (logid:24eb8b9b) HA health check task is running... 2025-06-29 03:22:46,332 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8db25fa0]) (logid:79be4a5c) ClusterDRS.poll is being called at 2025-06-29T01:23:00+0000 2025-06-29 03:22:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8db25fa0]) (logid:79be4a5c) Removed 0 old drs migration plans 2025-06-29 03:22:47,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:22:47,212 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:22:47,213 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:22:47,215 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:22:47,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd1b21eb]) (logid:a0b08edd) No inactive management server node found 2025-06-29 03:22:47,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd1b21eb]) (logid:a0b08edd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:48,830 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-428f0830]) (logid:043ab700) No inactive management server node found 2025-06-29 03:22:48,830 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-428f0830]) (logid:043ab700) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:50,164 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8142: Seq 7-8142: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:50,185 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8142 2025-06-29 03:22:50,252 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9079a795]) (logid:15276405) HA health check task is running... 2025-06-29 03:22:50,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf843d8c]) (logid:3bcb4633) No inactive management server node found 2025-06-29 03:22:50,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf843d8c]) (logid:3bcb4633) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f215d5ba]) (logid:69026808) No inactive management server node found 2025-06-29 03:22:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f215d5ba]) (logid:69026808) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45a2e64d]) (logid:4a576f55) No inactive management server node found 2025-06-29 03:22:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45a2e64d]) (logid:4a576f55) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:54,253 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-baddf053]) (logid:4298dcca) HA health check task is running... 2025-06-29 03:22:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d01ca85e]) (logid:45257543) No inactive management server node found 2025-06-29 03:22:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d01ca85e]) (logid:45257543) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:55,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8143: Seq 7-8143: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:22:55,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8143 2025-06-29 03:22:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04a8d67b]) (logid:9e806e50) No inactive management server node found 2025-06-29 03:22:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04a8d67b]) (logid:9e806e50) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:56,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3e21dc02]) (logid:8f58bf9e) Enabled non-edge zones available for scan: 1 2025-06-29 03:22:56,764 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3e21dc02]) (logid:8f58bf9e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:22:56,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3e21dc02]) (logid:8f58bf9e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:22:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c930b9d8]) (logid:c82f2b39) Enabled non-edge zones available for scan: 1 2025-06-29 03:22:56,897 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c930b9d8]) (logid:c82f2b39) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:22:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5328a6a3]) (logid:c4bba48e) No inactive management server node found 2025-06-29 03:22:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5328a6a3]) (logid:c4bba48e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:22:57,946 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-cd4885f4]) (logid:4b1d0773) AutoScaling Monitor is running... 2025-06-29 03:22:58,074 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-e44957a8]) (logid:51745685) Starting to remove all volume_stats rows older than [Sat Jun 28 15:22:58 CEST 2025]. 2025-06-29 03:22:58,074 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-e44957a8]) (logid:51745685) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:22:58 CEST 2025]. 2025-06-29 03:22:58,255 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-817319fd]) (logid:e18aaa3b) HA health check task is running... 2025-06-29 03:22:59,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c297e5]) (logid:40b2ab2c) No inactive management server node found 2025-06-29 03:22:59,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c297e5]) (logid:40b2ab2c) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:23:00,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfff24b3]) (logid:3dc935ba) No inactive management server node found 2025-06-29 03:23:00,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfff24b3]) (logid:3dc935ba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:02,143 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) StorageCollector is running... 2025-06-29 03:23:02,148 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:23:02,148 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:23:02,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:23:02,150 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-6717118844223095498: Routed from 90520738888109 2025-06-29 03:23:02,208 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 4-6717118844223095498: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:23:02,212 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:23:02,212 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:23:02,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:23:02,213 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-1612007191621930050: Routed from 90520738888109 2025-06-29 03:23:02,256 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e593869d]) (logid:c46f92cf) HA health check task is running... 2025-06-29 03:23:02,264 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-1612007191621930050: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:23:02,287 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:23:02,287 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:23:02,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:23:02,288 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-1612007191621930051: Routed from 90520738888109 2025-06-29 03:23:02,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-812cf85a]) (logid:65d2bfdd) No inactive management server node found 2025-06-29 03:23:02,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-812cf85a]) (logid:65d2bfdd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:02,381 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-1612007191621930051: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:23:02,392 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:23:02,392 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:23:02,392 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:23:02,393 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-3048092522799696763: Routed from 90520738888109 2025-06-29 03:23:03,443 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 1-3048092522799696763: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:23:03,445 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:23:03,446 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:23:03,446 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:23:03,447 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-1612007191621930052: Routed from 90520738888109 2025-06-29 03:23:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ea46065]) (logid:ff7b3141) No inactive management server node found 2025-06-29 03:23:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ea46065]) (logid:ff7b3141) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:04,506 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-19a27298]) (logid:52bd4e55) Seq 2-1612007191621930052: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:23:05,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8144: Seq 7-8144: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:23:05,186 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8144 2025-06-29 03:23:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acf4162a]) (logid:832fd95a) No inactive management server node found 2025-06-29 03:23:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acf4162a]) (logid:832fd95a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:05,548 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:23:05,552 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:23:05,552 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) Seq 2-3048092522799696764: Routed from 90520738888109 2025-06-29 03:23:05,565 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) Seq 1-3048092522799696764: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:23:05,611 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:23:05,612 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) Seq 2-1612007191621930053: Routed from 90520738888109 2025-06-29 03:23:05,662 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-2608cb27]) (logid:d6d4fde5) Seq 2-1612007191621930053: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:23:06,257 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a3d2e09c]) (logid:12b63f47) HA health check task is running... 2025-06-29 03:23:06,897 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e047c3ee]) (logid:70bf996f) No inactive management server node found 2025-06-29 03:23:06,897 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e047c3ee]) (logid:70bf996f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbe4b297]) (logid:06ef1aa9) No inactive management server node found 2025-06-29 03:23:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbe4b297]) (logid:06ef1aa9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:09,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b138cdc2]) (logid:40df8193) No inactive management server node found 2025-06-29 03:23:09,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b138cdc2]) (logid:40df8193) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:10,259 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2679bc2e]) (logid:f58f2dd5) HA health check task is running... 2025-06-29 03:23:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-24b4a65e]) (logid:328125b0) Found 0 routers to update status. 2025-06-29 03:23:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-24b4a65e]) (logid:328125b0) Found 0 VPC's to update Redundant State. 2025-06-29 03:23:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-24b4a65e]) (logid:328125b0) Found 0 networks to update RvR status. 2025-06-29 03:23:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfa4c68f]) (logid:89366ee1) No inactive management server node found 2025-06-29 03:23:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfa4c68f]) (logid:89366ee1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d72063d]) (logid:75418bc9) No inactive management server node found 2025-06-29 03:23:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d72063d]) (logid:75418bc9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:14,051 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-f925934f]) (logid:9ae330e5) Starting to remove all vm_stats rows older than [Sat Jun 28 15:23:14 CEST 2025]. 2025-06-29 03:23:14,086 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-f925934f]) (logid:9ae330e5) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:23:14 CEST 2025]. 2025-06-29 03:23:14,260 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b820683d]) (logid:9d1d71ce) HA health check task is running... 2025-06-29 03:23:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd9a1447]) (logid:c5b595b5) No inactive management server node found 2025-06-29 03:23:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd9a1447]) (logid:c5b595b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:15,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8145: Seq 7-8145: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:23:15,186 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8145 2025-06-29 03:23:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84b5db00]) (logid:9e314a68) No inactive management server node found 2025-06-29 03:23:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84b5db00]) (logid:9e314a68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b2386b5]) (logid:1fbefb71) No inactive management server node found 2025-06-29 03:23:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b2386b5]) (logid:1fbefb71) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:18,261 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4b2c91bb]) (logid:daceca4c) HA health check task is running... 2025-06-29 03:23:18,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66e1902c]) (logid:49056861) No inactive management server node found 2025-06-29 03:23:18,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66e1902c]) (logid:49056861) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-414516e2]) (logid:488ca6f2) No inactive management server node found 2025-06-29 03:23:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-414516e2]) (logid:488ca6f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5838e2b0]) (logid:27bd231f) No inactive management server node found 2025-06-29 03:23:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5838e2b0]) (logid:27bd231f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:22,262 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-068b0041]) (logid:edebd491) HA health check task is running... 2025-06-29 03:23:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9c08eb6]) (logid:25c8af10) No inactive management server node found 2025-06-29 03:23:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9c08eb6]) (logid:25c8af10) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:24,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08f4a591]) (logid:460dc65c) No inactive management server node found 2025-06-29 03:23:24,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08f4a591]) (logid:460dc65c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:25,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8146: Seq 7-8146: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:23:25,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8146 2025-06-29 03:23:26,264 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-48616c2f]) (logid:a33da8c0) HA health check task is running... 2025-06-29 03:23:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d583cc8]) (logid:65efb709) No inactive management server node found 2025-06-29 03:23:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d583cc8]) (logid:65efb709) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d373b32b]) (logid:404b9791) Enabled non-edge zones available for scan: 1 2025-06-29 03:23:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d373b32b]) (logid:404b9791) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:23:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d373b32b]) (logid:404b9791) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:23:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d76a96b4]) (logid:f18eb31e) Enabled non-edge zones available for scan: 1 2025-06-29 03:23:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d76a96b4]) (logid:f18eb31e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:23:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a284b25]) (logid:fe8586e8) No inactive management server node found 2025-06-29 03:23:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a284b25]) (logid:fe8586e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2700149e]) (logid:160ec0a7) No inactive management server node found 2025-06-29 03:23:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2700149e]) (logid:160ec0a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:30,265 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c88febb3]) (logid:ac9d8c52) HA health check task is running... 2025-06-29 03:23:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7f02f48]) (logid:94b88823) No inactive management server node found 2025-06-29 03:23:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7f02f48]) (logid:94b88823) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:30,871 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:23:30,872 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:23:30,873 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) Seq 2-3048092522799696765: Routed from 90520738888109 2025-06-29 03:23:30,916 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) Seq 1-3048092522799696765: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:23:30,918 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:23:30,918 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) Seq 2-1612007191621930054: Routed from 90520738888109 2025-06-29 03:23:30,921 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-32abda4d]) (logid:9611bdbe) Seq 2-1612007191621930054: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:23:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-584b150f]) (logid:c291bf6f) No inactive management server node found 2025-06-29 03:23:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-584b150f]) (logid:c291bf6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dd8922d]) (logid:b16bbaea) No inactive management server node found 2025-06-29 03:23:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dd8922d]) (logid:b16bbaea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:23:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:23:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:23:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:23:34,266 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e7e67729]) (logid:3c326503) HA health check task is running... 2025-06-29 03:23:35,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8147: Seq 7-8147: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:23:35,187 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8147 2025-06-29 03:23:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee813fdf]) (logid:1ec4325c) No inactive management server node found 2025-06-29 03:23:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee813fdf]) (logid:1ec4325c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ee88336]) (logid:f521903a) No inactive management server node found 2025-06-29 03:23:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ee88336]) (logid:f521903a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:38,268 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1841bc97]) (logid:b8f5466f) HA health check task is running... 2025-06-29 03:23:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55a79eb8]) (logid:ea752145) No inactive management server node found 2025-06-29 03:23:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55a79eb8]) (logid:ea752145) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:39,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b9597ba]) (logid:e84b2d41) No inactive management server node found 2025-06-29 03:23:39,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b9597ba]) (logid:e84b2d41) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:23:40,816 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-10a76109]) (logid:03989fb3) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:23:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-10a76109]) (logid:03989fb3) Snapshot scheduler is being called at [2025-06-29T01:23:40+0000]. 2025-06-29 03:23:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-10a76109]) (logid:03989fb3) There are [0] scheduled snapshots to be executed at [2025-06-29T01:23:40+0000]. 2025-06-29 03:23:40,916 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-b0e31612]) (logid:1721c8bd) External devices are not used. Skipping external device usage collection 2025-06-29 03:23:40,979 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-1984f023]) (logid:6c139971) Backup backup.poll is being called at 2025-06-29T01:23:40+0000 2025-06-29 03:23:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-cd363229]) (logid:32cf4a14) Found 0 running routers. 2025-06-29 03:23:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3a5b3747]) (logid:66be9091) Found 0 routers to update status. 2025-06-29 03:23:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3a5b3747]) (logid:66be9091) Found 0 VPC's to update Redundant State. 2025-06-29 03:23:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3a5b3747]) (logid:66be9091) Found 0 networks to update RvR status. 2025-06-29 03:23:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-362a37bc]) (logid:1efcda6e) DbCollector is running... 2025-06-29 03:23:41,286 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-bf731fc6]) (logid:e25e334a) Started resource counters recalculation periodic task. 2025-06-29 03:23:41,289 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-bf731fc6]) (logid:e25e334a) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:23:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e280de]) (logid:0465221e) No inactive management server node found 2025-06-29 03:23:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e280de]) (logid:0465221e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:41,639 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-bf731fc6]) (logid:e25e334a) Finished resource counters recalculation periodic task. 2025-06-29 03:23:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) ManagementServerCollector is running... 2025-06-29 03:23:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) DB properties were already loaded 2025-06-29 03:23:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:23:42,168 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Execution of process [4063627] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:23:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Exit value of process [4063627] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:23:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Process [4063627] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:23:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) usage install: null 2025-06-29 03:23:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Sessions active: 0 2025-06-29 03:23:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:23:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063630] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:23:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:23:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063635] for command [/bin/bash -c uname -r ]. 2025-06-29 03:23:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:23:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063636] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:23:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) system memory from /proc: 4887515136 2025-06-29 03:23:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:23:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063641] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:23:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) free memory from /proc: 509382656 2025-06-29 03:23:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:23:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063646] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:23:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) used memory from /proc: 769032 2025-06-29 03:23:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:23:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063647] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:23:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:23:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063648] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:23:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:23:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063649] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:23:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:23:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063656] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:23:42,236 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:23:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Successfully executed process [4063657] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail buffersdirect.count, 120 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail buffersdirect.used, 658750 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail buffersmapped.count, 7 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail buffersmapped.used, 798602 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail gcPS-Scavenge.count, 516 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail gcPS-Scavenge.time, 3753 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail jvmuptime, 44162321 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memoryheap.committed, 358088704 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memoryheap.init, 77594624 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memoryheap.usage, 0.11183294743111225 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorynon-heap.max, -1 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.PS-Eden-Space.usage, 0.041137730176450636 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.PS-Old-Gen.usage, 0.12586323623908652 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorypools.PS-Survivor-Space.usage, 0.3665935809795673 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorytotal.committed, 583204864 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorytotal.init, 85262336 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorytotal.max, 1908932607 2025-06-29 03:23:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail memorytotal.used, 434274808 2025-06-29 03:23:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail threadsdeadlocks, [] 2025-06-29 03:23:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail threadsnew.count, 0 2025-06-29 03:23:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:23:42,247 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-873e615b]) (logid:853cce89) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:23:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":145362216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44162246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433470440,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213481560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":509382656,"systemMemoryUsed":769032,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375125,53814734,4588358816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:23:42,248 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1471, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:23:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":145362216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44162246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433470440,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213481560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":509382656,"systemMemoryUsed":769032,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375125,53814734,4588358816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:23:42,248 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:23:42,253 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:23:42,253 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1471, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:23:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":145362216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44162246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433470440,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213481560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":509382656,"systemMemoryUsed":769032,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375125,53814734,4588358816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:23:42,254 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-330:[ctx-51b02dec]) (logid:dc4617af) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:23:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":145362216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44162246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433470440,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213481560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":509382656,"systemMemoryUsed":769032,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375125,53814734,4588358816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:23:42,269 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4bd0fae0]) (logid:08b812c9) HA health check task is running... 2025-06-29 03:23:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d90cacf2]) (logid:1e4e168c) No inactive management server node found 2025-06-29 03:23:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d90cacf2]) (logid:1e4e168c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e11f64c]) (logid:acaa7578) No inactive management server node found 2025-06-29 03:23:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e11f64c]) (logid:acaa7578) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:45,167 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8148: Seq 7-8148: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:23:45,188 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8148 2025-06-29 03:23:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:23:45,497 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:23:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec1f370c]) (logid:dfbd2ce0) No inactive management server node found 2025-06-29 03:23:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec1f370c]) (logid:dfbd2ce0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1812ffe2]) (logid:aa2ccff6) VM scheduler.poll is being called at 2025-06-29T01:24:00+0000 2025-06-29 03:23:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1812ffe2]) (logid:aa2ccff6) Got 0 scheduled jobs to be executed at 2025-06-29T01:24:00+0000 2025-06-29 03:23:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1812ffe2]) (logid:aa2ccff6) Cleaned up 0 VM scheduled job entries 2025-06-29 03:23:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-940575c2]) (logid:ac289dc7) VM Operation Thread Running 2025-06-29 03:23:46,189 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-3:[ctx-32683110]) (logid:5ef32122) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:23:46,270 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a7739b44]) (logid:128271f8) HA health check task is running... 2025-06-29 03:23:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4282fa19]) (logid:07c1e3af) ClusterDRS.poll is being called at 2025-06-29T01:24:00+0000 2025-06-29 03:23:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4282fa19]) (logid:07c1e3af) Removed 0 old drs migration plans 2025-06-29 03:23:47,055 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c6668d02]) (logid:bdcea5d0) Resetting hosts suitable for reconnect 2025-06-29 03:23:47,061 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c6668d02]) (logid:bdcea5d0) Completed resetting hosts suitable for reconnect 2025-06-29 03:23:47,061 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c6668d02]) (logid:bdcea5d0) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:23:47,062 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c6668d02]) (logid:bdcea5d0) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:23:47,062 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c6668d02]) (logid:bdcea5d0) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:23:47,063 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c6668d02]) (logid:bdcea5d0) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:23:47,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:23:47,212 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:23:47,213 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:23:47,216 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:23:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58d26795]) (logid:320343bd) No inactive management server node found 2025-06-29 03:23:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58d26795]) (logid:320343bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:48,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-129b6206]) (logid:45ef260a) No inactive management server node found 2025-06-29 03:23:48,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-129b6206]) (logid:45ef260a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:50,271 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2738aa50]) (logid:0e46dd0a) HA health check task is running... 2025-06-29 03:23:50,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0133633]) (logid:7fafc752) No inactive management server node found 2025-06-29 03:23:50,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0133633]) (logid:7fafc752) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a768e83]) (logid:7630f550) No inactive management server node found 2025-06-29 03:23:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a768e83]) (logid:7630f550) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d39e99e]) (logid:6e925f01) No inactive management server node found 2025-06-29 03:23:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d39e99e]) (logid:6e925f01) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:54,272 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0e89d138]) (logid:fece3a74) HA health check task is running... 2025-06-29 03:23:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1c54f41]) (logid:cf6c2f15) No inactive management server node found 2025-06-29 03:23:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1c54f41]) (logid:cf6c2f15) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:55,167 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8150: Seq 7-8150: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:23:55,188 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8150 2025-06-29 03:23:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-305c76d9]) (logid:f28bf289) No inactive management server node found 2025-06-29 03:23:56,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-305c76d9]) (logid:f28bf289) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:23:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c5ba898c]) (logid:af19c3b0) Enabled non-edge zones available for scan: 1 2025-06-29 03:23:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c5ba898c]) (logid:af19c3b0) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:23:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c5ba898c]) (logid:af19c3b0) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:23:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-24969509]) (logid:4cff01ef) Enabled non-edge zones available for scan: 1 2025-06-29 03:23:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-24969509]) (logid:4cff01ef) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:23:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98b0162b]) (logid:5eedaf6e) No inactive management server node found 2025-06-29 03:23:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98b0162b]) (logid:5eedaf6e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:23:57,948 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-6a8d3a8f]) (logid:c8407ba6) AutoScaling Monitor is running... 2025-06-29 03:23:58,076 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-fb443f6e]) (logid:86ff47f2) Starting to remove all volume_stats rows older than [Sat Jun 28 15:23:58 CEST 2025]. 2025-06-29 03:23:58,077 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-fb443f6e]) (logid:86ff47f2) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:23:58 CEST 2025]. 2025-06-29 03:23:58,273 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e524ce4b]) (logid:93b5f9e2) HA health check task is running... 2025-06-29 03:23:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-797b46a1]) (logid:d4949b4a) No inactive management server node found 2025-06-29 03:23:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-797b46a1]) (logid:d4949b4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2859794f]) (logid:f0ccb1e6) No inactive management server node found 2025-06-29 03:24:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2859794f]) (logid:f0ccb1e6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:02,275 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7d9c2bae]) (logid:26cec204) HA health check task is running... 2025-06-29 03:24:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a24a08fe]) (logid:6007811f) No inactive management server node found 2025-06-29 03:24:02,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a24a08fe]) (logid:6007811f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cefa7c4f]) (logid:dc96643c) No inactive management server node found 2025-06-29 03:24:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cefa7c4f]) (logid:dc96643c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:04,506 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) StorageCollector is running... 2025-06-29 03:24:04,512 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:24:04,512 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:24:04,513 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:24:04,515 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 2-6717118844223095499: Routed from 90520738888109 2025-06-29 03:24:04,591 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 4-6717118844223095499: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:24:04,595 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:24:04,595 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:24:04,595 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:24:04,596 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 2-3048092522799696766: Routed from 90520738888109 2025-06-29 03:24:04,644 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 1-3048092522799696766: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:24:04,666 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:24:04,666 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:24:04,666 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:24:04,667 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 2-3048092522799696767: Routed from 90520738888109 2025-06-29 03:24:04,756 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 1-3048092522799696767: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:24:04,766 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:24:04,767 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:24:04,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:24:04,767 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 2-3048092522799696768: Routed from 90520738888109 2025-06-29 03:24:05,167 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8151: Seq 7-8151: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:24:05,188 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8151 2025-06-29 03:24:05,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07fe50c7]) (logid:d0efe1f5) No inactive management server node found 2025-06-29 03:24:05,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07fe50c7]) (logid:d0efe1f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:05,718 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:24:05,721 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:24:05,722 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) Seq 2-3048092522799696769: Routed from 90520738888109 2025-06-29 03:24:05,775 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) Seq 1-3048092522799696769: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:24:05,817 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 1-3048092522799696768: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:24:05,819 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:24:05,819 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:24:05,819 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:24:05,820 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 2-1612007191621930055: Routed from 90520738888109 2025-06-29 03:24:05,821 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:24:05,822 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) Seq 2-1612007191621930056: Routed from 90520738888109 2025-06-29 03:24:05,870 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-857056b1]) (logid:6cce4909) Seq 2-1612007191621930056: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:24:06,276 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a0b1911c]) (logid:472ccefe) HA health check task is running... 2025-06-29 03:24:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f76b68]) (logid:1897d33f) No inactive management server node found 2025-06-29 03:24:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f76b68]) (logid:1897d33f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:06,839 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1d2ef1f4]) (logid:dc553457) Seq 2-1612007191621930055: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:24:08,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5865bd41]) (logid:734df2b3) No inactive management server node found 2025-06-29 03:24:08,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5865bd41]) (logid:734df2b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:09,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efbacd2d]) (logid:8d8209ed) No inactive management server node found 2025-06-29 03:24:09,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efbacd2d]) (logid:8d8209ed) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:10,277 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5772956f]) (logid:b9ae87c1) HA health check task is running... 2025-06-29 03:24:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-91bce3bb]) (logid:7b7b223e) Found 0 routers to update status. 2025-06-29 03:24:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-91bce3bb]) (logid:7b7b223e) Found 0 VPC's to update Redundant State. 2025-06-29 03:24:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-91bce3bb]) (logid:7b7b223e) Found 0 networks to update RvR status. 2025-06-29 03:24:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Running Capacity Checker ... 2025-06-29 03:24:11,069 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) recalculating system capacity 2025-06-29 03:24:11,069 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Executing cpu/ram capacity update 2025-06-29 03:24:11,072 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:11,073 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:11,079 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:11,083 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:24:11,083 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:24:11,083 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:24:11,105 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:11,106 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:11,110 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:11,111 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:24:11,111 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:24:11,112 DEBUG [c.c.c.CapacityManagerImpl] (pool-322-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:24:11,120 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Done executing cpu/ram capacity update 2025-06-29 03:24:11,120 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Executing storage capacity update 2025-06-29 03:24:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:24:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:24:11,126 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:24:11,128 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:24:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:24:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:24:11,133 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:24:11,135 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:24:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:24:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:24:11,140 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:24:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:24:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:24:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:24:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:24:11,155 DEBUG [c.c.s.StorageManagerImpl] (pool-323-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:24:11,155 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Done executing storage capacity update 2025-06-29 03:24:11,156 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Executing capacity updates for public ip and Vlans 2025-06-29 03:24:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Done capacity updates for public ip and Vlans 2025-06-29 03:24:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Executing capacity updates for private ip 2025-06-29 03:24:11,173 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Done executing capacity updates for private ip 2025-06-29 03:24:11,173 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Done recalculating system capacity 2025-06-29 03:24:11,204 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-3b2fed78]) (logid:5aa8208f) Done running Capacity Checker ... 2025-06-29 03:24:11,261 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-e243c70e]) (logid:62a4bda4) Checking if any host reservation can be released ... 2025-06-29 03:24:11,266 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-e243c70e]) (logid:62a4bda4) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:11,270 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-e243c70e]) (logid:62a4bda4) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:11,270 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-e243c70e]) (logid:62a4bda4) Done running HostReservationReleaseChecker ... 2025-06-29 03:24:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ff3f073]) (logid:ae897b4c) No inactive management server node found 2025-06-29 03:24:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ff3f073]) (logid:ae897b4c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:12,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e8441e8]) (logid:4aab3991) No inactive management server node found 2025-06-29 03:24:12,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e8441e8]) (logid:4aab3991) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:14,087 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6cdcff3c]) (logid:52793b21) Starting to remove all vm_stats rows older than [Sat Jun 28 15:24:14 CEST 2025]. 2025-06-29 03:24:14,109 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6cdcff3c]) (logid:52793b21) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:24:14 CEST 2025]. 2025-06-29 03:24:14,278 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8af2f8d7]) (logid:fb2f54ad) HA health check task is running... 2025-06-29 03:24:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54959f50]) (logid:a09f0265) No inactive management server node found 2025-06-29 03:24:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54959f50]) (logid:a09f0265) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:15,168 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8152: Seq 7-8152: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:24:15,189 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8152 2025-06-29 03:24:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b926881c]) (logid:0b9bf697) No inactive management server node found 2025-06-29 03:24:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b926881c]) (logid:0b9bf697) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:17,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdb9efad]) (logid:fc24cdb3) No inactive management server node found 2025-06-29 03:24:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdb9efad]) (logid:fc24cdb3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:18,279 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-713dd3a9]) (logid:d11dd9db) HA health check task is running... 2025-06-29 03:24:18,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbcae2b9]) (logid:ce2595de) No inactive management server node found 2025-06-29 03:24:18,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbcae2b9]) (logid:ce2595de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:24:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84769269]) (logid:0020ee61) No inactive management server node found 2025-06-29 03:24:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84769269]) (logid:0020ee61) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e5155e2]) (logid:27a4b932) No inactive management server node found 2025-06-29 03:24:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e5155e2]) (logid:27a4b932) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:22,281 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e8b70cb2]) (logid:f8ee5c56) HA health check task is running... 2025-06-29 03:24:23,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c221c77c]) (logid:b958f209) No inactive management server node found 2025-06-29 03:24:23,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c221c77c]) (logid:b958f209) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e7b28e4]) (logid:c2068465) No inactive management server node found 2025-06-29 03:24:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e7b28e4]) (logid:c2068465) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:24:25,168 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8153: Seq 7-8153: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:24:25,189 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8153 2025-06-29 03:24:26,282 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-51e24862]) (logid:42e8e805) HA health check task is running... 2025-06-29 03:24:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dd5f7f6]) (logid:e208ea8f) No inactive management server node found 2025-06-29 03:24:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dd5f7f6]) (logid:e208ea8f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:24:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b660bfbd]) (logid:7bdf8c19) Enabled non-edge zones available for scan: 1 2025-06-29 03:24:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b660bfbd]) (logid:7bdf8c19) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:24:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b660bfbd]) (logid:7bdf8c19) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:24:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-15553518]) (logid:f88d68c5) Enabled non-edge zones available for scan: 1 2025-06-29 03:24:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-15553518]) (logid:f88d68c5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:24:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283908d9]) (logid:341fe4d4) No inactive management server node found 2025-06-29 03:24:27,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283908d9]) (logid:341fe4d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d406dbb]) (logid:7416c536) No inactive management server node found 2025-06-29 03:24:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d406dbb]) (logid:7416c536) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:30,284 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ee6adddd]) (logid:edd33863) HA health check task is running... 2025-06-29 03:24:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95fe83c8]) (logid:d0b3845a) No inactive management server node found 2025-06-29 03:24:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95fe83c8]) (logid:d0b3845a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:30,924 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:24:30,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:24:30,925 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) Seq 2-3048092522799696770: Routed from 90520738888109 2025-06-29 03:24:30,968 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) Seq 1-3048092522799696770: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:24:30,969 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:24:30,970 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) Seq 2-1612007191621930057: Routed from 90520738888109 2025-06-29 03:24:30,973 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-2b08206c]) (logid:d6563120) Seq 2-1612007191621930057: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:24:32,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0da7afa]) (logid:4cfaf26f) No inactive management server node found 2025-06-29 03:24:32,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0da7afa]) (logid:4cfaf26f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-373bfe2a]) (logid:fe0528f3) No inactive management server node found 2025-06-29 03:24:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-373bfe2a]) (logid:fe0528f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:33,986 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:33,988 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:33,989 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:24:33,991 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:24:34,285 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-252e3bd2]) (logid:fe477b5e) HA health check task is running... 2025-06-29 03:24:35,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8154: Seq 7-8154: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:24:35,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8154 2025-06-29 03:24:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bfb09d0]) (logid:f897b785) No inactive management server node found 2025-06-29 03:24:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bfb09d0]) (logid:f897b785) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-184c5aa1]) (logid:6ec1daf9) No inactive management server node found 2025-06-29 03:24:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-184c5aa1]) (logid:6ec1daf9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:38,287 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a4ba7e79]) (logid:4b3ded1d) HA health check task is running... 2025-06-29 03:24:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40fed8cc]) (logid:621805f1) No inactive management server node found 2025-06-29 03:24:38,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40fed8cc]) (logid:621805f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7adb9ada]) (logid:28835021) No inactive management server node found 2025-06-29 03:24:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7adb9ada]) (logid:28835021) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fcc2e8fe]) (logid:e7626d23) Found 0 routers to update status. 2025-06-29 03:24:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fcc2e8fe]) (logid:e7626d23) Found 0 VPC's to update Redundant State. 2025-06-29 03:24:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fcc2e8fe]) (logid:e7626d23) Found 0 networks to update RvR status. 2025-06-29 03:24:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-5e0ac241]) (logid:8ec9827a) DbCollector is running... 2025-06-29 03:24:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7537bdc9]) (logid:3dbcf7e4) No inactive management server node found 2025-06-29 03:24:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7537bdc9]) (logid:3dbcf7e4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) ManagementServerCollector is running... 2025-06-29 03:24:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) DB properties were already loaded 2025-06-29 03:24:42,154 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:24:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Execution of process [4063675] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:24:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Exit value of process [4063675] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:24:42,168 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Process [4063675] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:24:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) usage install: null 2025-06-29 03:24:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Sessions active: 0 2025-06-29 03:24:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:24:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063678] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:24:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:24:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063683] for command [/bin/bash -c uname -r ]. 2025-06-29 03:24:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:24:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063684] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:24:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) system memory from /proc: 4887515136 2025-06-29 03:24:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:24:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063689] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:24:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) free memory from /proc: 500867072 2025-06-29 03:24:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:24:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063694] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:24:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) used memory from /proc: 769160 2025-06-29 03:24:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:24:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063695] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:24:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:24:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063696] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:24:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:24:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063697] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:24:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:24:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063704] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:24:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:24:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Successfully executed process [4063705] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail buffersdirect.count, 120 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail buffersdirect.used, 658750 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail buffersmapped.count, 7 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail buffersmapped.used, 798602 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail gcPS-Scavenge.count, 517 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail gcPS-Scavenge.time, 3758 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail jvmuptime, 44222322 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memoryheap.committed, 358612992 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memoryheap.init, 77594624 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memoryheap.usage, 0.11169039656322954 2025-06-29 03:24:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorynon-heap.max, -1 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.PS-Eden-Space.usage, 0.040725359368306124 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.PS-Old-Gen.usage, 0.12596622049393824 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorypools.PS-Survivor-Space.usage, 0.35457435021033656 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorytotal.committed, 583729152 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorytotal.init, 85262336 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorytotal.max, 1908932607 2025-06-29 03:24:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail memorytotal.used, 433971056 2025-06-29 03:24:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail threadsdeadlocks, [] 2025-06-29 03:24:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail threadsnew.count, 0 2025-06-29 03:24:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:24:42,249 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-1cbc2fd6]) (logid:e277c153) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:24:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":146155656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44222246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433176648,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213209440,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500867072,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375456,53815049,4588381937],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:24:42,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1473, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:24:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":146155656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44222246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433176648,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213209440,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500867072,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375456,53815049,4588381937],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:24:42,249 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:24:42,254 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-330:[ctx-033e66b8]) (logid:96653bdc) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:24:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":146155656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44222246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433176648,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213209440,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500867072,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375456,53815049,4588381937],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:24:42,254 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:24:42,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1473, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:24:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":146155656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44222246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":433176648,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":213209440,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":500867072,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.11,0.09],"systemCyclesUsage":[66375456,53815049,4588381937],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:24:42,288 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-eb88d08f]) (logid:acd739c1) HA health check task is running... 2025-06-29 03:24:42,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b283c6b6]) (logid:12f2d11b) No inactive management server node found 2025-06-29 03:24:42,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b283c6b6]) (logid:12f2d11b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61998244]) (logid:73019c2a) No inactive management server node found 2025-06-29 03:24:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61998244]) (logid:73019c2a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:45,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8155: Seq 7-8155: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:24:45,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8155 2025-06-29 03:24:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:24:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:24:45,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-620f8d50]) (logid:79d49302) No inactive management server node found 2025-06-29 03:24:45,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-620f8d50]) (logid:79d49302) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-72c84e1b]) (logid:1763d97d) VM scheduler.poll is being called at 2025-06-29T01:25:00+0000 2025-06-29 03:24:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-72c84e1b]) (logid:1763d97d) Got 0 scheduled jobs to be executed at 2025-06-29T01:25:00+0000 2025-06-29 03:24:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-72c84e1b]) (logid:1763d97d) Cleaned up 0 VM scheduled job entries 2025-06-29 03:24:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-1072780f]) (logid:a82c21a0) VM Operation Thread Running 2025-06-29 03:24:46,289 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cfacf797]) (logid:921ebe47) HA health check task is running... 2025-06-29 03:24:46,334 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1f9e4d85]) (logid:0ac6a267) ClusterDRS.poll is being called at 2025-06-29T01:25:00+0000 2025-06-29 03:24:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1f9e4d85]) (logid:0ac6a267) Removed 0 old drs migration plans 2025-06-29 03:24:47,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:47,211 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:47,213 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:24:47,216 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:24:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a261e21b]) (logid:8a21d331) No inactive management server node found 2025-06-29 03:24:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a261e21b]) (logid:8a21d331) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:24:48,800 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:24:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d05b9bb0]) (logid:6dc8f1a2) No inactive management server node found 2025-06-29 03:24:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d05b9bb0]) (logid:6dc8f1a2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:50,290 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8e29c9af]) (logid:2e9aeac4) HA health check task is running... 2025-06-29 03:24:50,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58ecbe02]) (logid:d829dbcb) No inactive management server node found 2025-06-29 03:24:50,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58ecbe02]) (logid:d829dbcb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95d93ce3]) (logid:5bf4186b) No inactive management server node found 2025-06-29 03:24:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95d93ce3]) (logid:5bf4186b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f6fd0c2]) (logid:191a051c) No inactive management server node found 2025-06-29 03:24:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f6fd0c2]) (logid:191a051c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:54,292 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e384a44b]) (logid:b370df0b) HA health check task is running... 2025-06-29 03:24:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d31cf11]) (logid:c5044b48) No inactive management server node found 2025-06-29 03:24:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d31cf11]) (logid:c5044b48) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:55,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8157: Seq 7-8157: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:24:55,191 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8157 2025-06-29 03:24:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-908ba000]) (logid:490b3655) No inactive management server node found 2025-06-29 03:24:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-908ba000]) (logid:490b3655) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8d3ffc5c]) (logid:e203c5fc) Enabled non-edge zones available for scan: 1 2025-06-29 03:24:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8d3ffc5c]) (logid:e203c5fc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:24:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8d3ffc5c]) (logid:e203c5fc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:24:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-078403e2]) (logid:d42f8ad3) Enabled non-edge zones available for scan: 1 2025-06-29 03:24:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-078403e2]) (logid:d42f8ad3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:24:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3159d1d1]) (logid:7ffa4f4b) No inactive management server node found 2025-06-29 03:24:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3159d1d1]) (logid:7ffa4f4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:24:57,951 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-ad3a0992]) (logid:51d81e65) AutoScaling Monitor is running... 2025-06-29 03:24:58,079 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-07337409]) (logid:e49ae15b) Starting to remove all volume_stats rows older than [Sat Jun 28 15:24:58 CEST 2025]. 2025-06-29 03:24:58,079 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-07337409]) (logid:e49ae15b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:24:58 CEST 2025]. 2025-06-29 03:24:58,293 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c4b11ed1]) (logid:48458cba) HA health check task is running... 2025-06-29 03:24:59,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee2c0df7]) (logid:004014c3) No inactive management server node found 2025-06-29 03:24:59,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee2c0df7]) (logid:004014c3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bf30e42]) (logid:b55ace2d) No inactive management server node found 2025-06-29 03:25:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bf30e42]) (logid:b55ace2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:02,294 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1935f83e]) (logid:69d39a1b) HA health check task is running... 2025-06-29 03:25:02,340 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dec924e]) (logid:0a5e5e38) No inactive management server node found 2025-06-29 03:25:02,340 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dec924e]) (logid:0a5e5e38) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:25:03,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de474ee5]) (logid:f189de1f) No inactive management server node found 2025-06-29 03:25:03,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de474ee5]) (logid:f189de1f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:05,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8158: Seq 7-8158: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:25:05,191 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8158 2025-06-29 03:25:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb5df72e]) (logid:468277d3) No inactive management server node found 2025-06-29 03:25:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb5df72e]) (logid:468277d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:05,924 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:25:05,928 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:25:05,929 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) Seq 2-3048092522799696771: Routed from 90520738888109 2025-06-29 03:25:05,983 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) Seq 1-3048092522799696771: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:25:06,037 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:25:06,038 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) Seq 2-1612007191621930058: Routed from 90520738888109 2025-06-29 03:25:06,096 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-34300582]) (logid:098d5847) Seq 2-1612007191621930058: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:25:06,295 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bff6e1d6]) (logid:3f51e32f) HA health check task is running... 2025-06-29 03:25:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b81f6116]) (logid:ca48888e) No inactive management server node found 2025-06-29 03:25:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b81f6116]) (logid:ca48888e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:06,840 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) StorageCollector is running... 2025-06-29 03:25:06,845 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:25:06,845 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:25:06,845 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:25:06,846 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-6717118844223095500: Routed from 90520738888109 2025-06-29 03:25:06,898 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 4-6717118844223095500: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:25:06,902 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:25:06,902 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:25:06,902 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:25:06,903 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-3048092522799696772: Routed from 90520738888109 2025-06-29 03:25:06,950 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 1-3048092522799696772: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:25:06,961 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:25:06,961 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:25:06,961 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:25:06,961 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-1612007191621930059: Routed from 90520738888109 2025-06-29 03:25:06,973 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-1612007191621930059: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:25:06,982 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:25:06,982 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:25:06,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:25:06,983 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-3048092522799696773: Routed from 90520738888109 2025-06-29 03:25:08,074 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 1-3048092522799696773: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:25:08,076 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:25:08,076 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:25:08,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:25:08,077 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-1612007191621930060: Routed from 90520738888109 2025-06-29 03:25:08,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b26702d]) (logid:c7076039) No inactive management server node found 2025-06-29 03:25:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b26702d]) (logid:c7076039) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:25:09,140 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3a03945d]) (logid:167eda82) Seq 2-1612007191621930060: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:25:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78294408]) (logid:0d6b1694) No inactive management server node found 2025-06-29 03:25:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78294408]) (logid:0d6b1694) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:10,297 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ebf56c37]) (logid:5c70aa20) HA health check task is running... 2025-06-29 03:25:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-46e2bd81]) (logid:a58a8b26) Found 0 routers to update status. 2025-06-29 03:25:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-46e2bd81]) (logid:a58a8b26) Found 0 VPC's to update Redundant State. 2025-06-29 03:25:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-46e2bd81]) (logid:a58a8b26) Found 0 networks to update RvR status. 2025-06-29 03:25:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6556a34f]) (logid:47c8d8db) No inactive management server node found 2025-06-29 03:25:11,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6556a34f]) (logid:47c8d8db) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a973e75]) (logid:8f9851f5) No inactive management server node found 2025-06-29 03:25:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a973e75]) (logid:8f9851f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:14,110 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-9a584624]) (logid:10e656f9) Starting to remove all vm_stats rows older than [Sat Jun 28 15:25:14 CEST 2025]. 2025-06-29 03:25:14,144 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-9a584624]) (logid:10e656f9) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:25:14 CEST 2025]. 2025-06-29 03:25:14,298 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-63e99690]) (logid:32158550) HA health check task is running... 2025-06-29 03:25:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5af196de]) (logid:2d5a7282) No inactive management server node found 2025-06-29 03:25:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5af196de]) (logid:2d5a7282) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:15,170 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8159: Seq 7-8159: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:25:15,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8159 2025-06-29 03:25:15,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a34d12e]) (logid:9a0bc9ea) No inactive management server node found 2025-06-29 03:25:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a34d12e]) (logid:9a0bc9ea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f8feae63]) (logid:e49d6c88) Resetting hosts suitable for reconnect 2025-06-29 03:25:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f8feae63]) (logid:e49d6c88) Completed resetting hosts suitable for reconnect 2025-06-29 03:25:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f8feae63]) (logid:e49d6c88) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:25:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f8feae63]) (logid:e49d6c88) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:25:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f8feae63]) (logid:e49d6c88) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:25:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f8feae63]) (logid:e49d6c88) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:25:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7197cc07]) (logid:1c5e1fd5) No inactive management server node found 2025-06-29 03:25:17,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7197cc07]) (logid:1c5e1fd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:18,299 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4065ec37]) (logid:498764d8) HA health check task is running... 2025-06-29 03:25:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fdd058c]) (logid:564aaba2) No inactive management server node found 2025-06-29 03:25:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fdd058c]) (logid:564aaba2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-300b9524]) (logid:f37042cc) No inactive management server node found 2025-06-29 03:25:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-300b9524]) (logid:f37042cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43a205f7]) (logid:61ba6fb7) No inactive management server node found 2025-06-29 03:25:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43a205f7]) (logid:61ba6fb7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:22,300 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-df7b70ca]) (logid:5dfd2889) HA health check task is running... 2025-06-29 03:25:23,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be8e8396]) (logid:05f14138) No inactive management server node found 2025-06-29 03:25:23,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be8e8396]) (logid:05f14138) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc744f5]) (logid:f0a7524e) No inactive management server node found 2025-06-29 03:25:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc744f5]) (logid:f0a7524e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:25,171 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8160: Seq 7-8160: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:25:25,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8160 2025-06-29 03:25:26,301 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-87e3f51d]) (logid:ab6c578b) HA health check task is running... 2025-06-29 03:25:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4344b623]) (logid:1204a54e) No inactive management server node found 2025-06-29 03:25:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4344b623]) (logid:1204a54e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3b0371cd]) (logid:73d2ca90) Enabled non-edge zones available for scan: 1 2025-06-29 03:25:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3b0371cd]) (logid:73d2ca90) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:25:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3b0371cd]) (logid:73d2ca90) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:25:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-02bfaacf]) (logid:358116db) Enabled non-edge zones available for scan: 1 2025-06-29 03:25:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-02bfaacf]) (logid:358116db) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:25:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6243fd84]) (logid:0bdb1f49) No inactive management server node found 2025-06-29 03:25:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6243fd84]) (logid:0bdb1f49) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:29,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d9083c7]) (logid:b1f22422) No inactive management server node found 2025-06-29 03:25:29,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d9083c7]) (logid:b1f22422) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:30,303 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8419d2c0]) (logid:ca703618) HA health check task is running... 2025-06-29 03:25:30,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42821298]) (logid:1a427925) No inactive management server node found 2025-06-29 03:25:30,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42821298]) (logid:1a427925) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:30,975 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:25:30,976 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:25:30,976 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) Seq 2-3048092522799696774: Routed from 90520738888109 2025-06-29 03:25:30,979 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) Seq 1-3048092522799696774: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:25:30,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:25:30,981 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) Seq 2-1612007191621930061: Routed from 90520738888109 2025-06-29 03:25:30,984 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e1e2367e]) (logid:d181f7f6) Seq 2-1612007191621930061: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:25:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-229255a9]) (logid:3a1fffde) No inactive management server node found 2025-06-29 03:25:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-229255a9]) (logid:3a1fffde) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:33,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-368db4b1]) (logid:f9243803) No inactive management server node found 2025-06-29 03:25:33,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-368db4b1]) (logid:f9243803) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:25:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:25:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:25:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:25:34,304 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-58922cfc]) (logid:a58c7349) HA health check task is running... 2025-06-29 03:25:35,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8161: Seq 7-8161: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:25:35,196 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8161 2025-06-29 03:25:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bc5f24a]) (logid:bb8b0f4e) No inactive management server node found 2025-06-29 03:25:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bc5f24a]) (logid:bb8b0f4e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:25:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e8ada64]) (logid:e39f93fb) No inactive management server node found 2025-06-29 03:25:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e8ada64]) (logid:e39f93fb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:38,305 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a2bb5d6f]) (logid:6b979ca0) HA health check task is running... 2025-06-29 03:25:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25d22da4]) (logid:5e53c373) No inactive management server node found 2025-06-29 03:25:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25d22da4]) (logid:5e53c373) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5337fa1]) (logid:1957087d) No inactive management server node found 2025-06-29 03:25:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5337fa1]) (logid:1957087d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-419b64de]) (logid:73e9e9ac) Found 0 routers to update status. 2025-06-29 03:25:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-419b64de]) (logid:73e9e9ac) Found 0 VPC's to update Redundant State. 2025-06-29 03:25:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-419b64de]) (logid:73e9e9ac) Found 0 networks to update RvR status. 2025-06-29 03:25:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-5e11faca]) (logid:f5cfeab2) DbCollector is running... 2025-06-29 03:25:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ebac4bd]) (logid:f4175363) No inactive management server node found 2025-06-29 03:25:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ebac4bd]) (logid:f4175363) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) ManagementServerCollector is running... 2025-06-29 03:25:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) DB properties were already loaded 2025-06-29 03:25:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:25:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Execution of process [4063714] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:25:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Exit value of process [4063714] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:25:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Process [4063714] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:25:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) usage install: null 2025-06-29 03:25:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Sessions active: 0 2025-06-29 03:25:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:25:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063717] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:25:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:25:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063722] for command [/bin/bash -c uname -r ]. 2025-06-29 03:25:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:25:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063723] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:25:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) system memory from /proc: 4887515136 2025-06-29 03:25:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:25:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063728] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:25:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) free memory from /proc: 496508928 2025-06-29 03:25:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:25:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063733] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:25:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) used memory from /proc: 769160 2025-06-29 03:25:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:25:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063734] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:25:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:25:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063735] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:25:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:25:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063736] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:25:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:25:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063743] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:25:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:25:42,241 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Successfully executed process [4063744] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail buffersdirect.count, 120 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail buffersdirect.used, 658750 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail buffersmapped.count, 7 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail buffersmapped.used, 798602 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail gcPS-Scavenge.count, 518 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail gcPS-Scavenge.time, 3763 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail jvmuptime, 44282320 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memoryheap.committed, 358612992 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memoryheap.init, 77594624 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memoryheap.usage, 0.10726056600527199 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorynon-heap.max, -1 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.PS-Eden-Space.usage, 0.02825830115313214 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.PS-Old-Gen.usage, 0.12664111600782965 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorypools.PS-Survivor-Space.usage, 0.29447232759915865 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorytotal.committed, 583729152 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorytotal.init, 85262336 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorytotal.max, 1908932607 2025-06-29 03:25:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail memorytotal.used, 425528840 2025-06-29 03:25:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail threadsdeadlocks, [] 2025-06-29 03:25:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail threadsnew.count, 0 2025-06-29 03:25:42,245 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:25:42,247 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-f6f997d7]) (logid:3543a1a5) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:25:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":154607864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44282246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":424724440,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":204753192,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496508928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.1,0.09],"systemCyclesUsage":[66375767,53815344,4588405073],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:25:42,247 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1475, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:25:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":154607864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44282246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":424724440,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":204753192,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496508928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.1,0.09],"systemCyclesUsage":[66375767,53815344,4588405073],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:25:42,247 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:25:42,250 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-330:[ctx-bc50b816]) (logid:2f7d03ab) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:25:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":154607864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44282246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":424724440,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":204753192,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496508928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.1,0.09],"systemCyclesUsage":[66375767,53815344,4588405073],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:25:42,251 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:25:42,251 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1475, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:25:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":154607864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44282246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":424724440,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":204753192,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":496508928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.1,0.09],"systemCyclesUsage":[66375767,53815344,4588405073],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:25:42,307 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-33302de6]) (logid:2a3af619) HA health check task is running... 2025-06-29 03:25:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1966cc25]) (logid:38b49538) No inactive management server node found 2025-06-29 03:25:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1966cc25]) (logid:38b49538) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e86a24d8]) (logid:002f94fa) No inactive management server node found 2025-06-29 03:25:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e86a24d8]) (logid:002f94fa) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:25:45,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8162: Seq 7-8162: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:25:45,195 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8162 2025-06-29 03:25:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:25:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:25:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ea449cc]) (logid:935c8c4f) No inactive management server node found 2025-06-29 03:25:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ea449cc]) (logid:935c8c4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-61515468]) (logid:e10a1411) VM scheduler.poll is being called at 2025-06-29T01:26:00+0000 2025-06-29 03:25:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-61515468]) (logid:e10a1411) Got 0 scheduled jobs to be executed at 2025-06-29T01:26:00+0000 2025-06-29 03:25:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-61515468]) (logid:e10a1411) Cleaned up 0 VM scheduled job entries 2025-06-29 03:25:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-dfde163a]) (logid:a827b529) VM Operation Thread Running 2025-06-29 03:25:46,308 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8cc4a677]) (logid:9cd3738c) HA health check task is running... 2025-06-29 03:25:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d072418f]) (logid:6e016c38) ClusterDRS.poll is being called at 2025-06-29T01:26:00+0000 2025-06-29 03:25:46,339 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d072418f]) (logid:6e016c38) Removed 0 old drs migration plans 2025-06-29 03:25:47,233 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:25:47,234 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:25:47,236 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:25:47,239 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:25:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9407ca4]) (logid:31536be2) No inactive management server node found 2025-06-29 03:25:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9407ca4]) (logid:31536be2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2d70dd4]) (logid:4d6530ff) No inactive management server node found 2025-06-29 03:25:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2d70dd4]) (logid:4d6530ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:50,309 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c1512ae0]) (logid:938e46a8) HA health check task is running... 2025-06-29 03:25:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-564c636a]) (logid:dcd31d51) No inactive management server node found 2025-06-29 03:25:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-564c636a]) (logid:dcd31d51) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:25:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65d8c831]) (logid:9764aba4) No inactive management server node found 2025-06-29 03:25:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65d8c831]) (logid:9764aba4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:53,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6bf7741]) (logid:adcb038c) No inactive management server node found 2025-06-29 03:25:53,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6bf7741]) (logid:adcb038c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:54,311 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-263465ec]) (logid:d0b6a84c) HA health check task is running... 2025-06-29 03:25:54,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2943fa5c]) (logid:41b49e56) No inactive management server node found 2025-06-29 03:25:54,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2943fa5c]) (logid:41b49e56) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:55,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8164: Seq 7-8164: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:25:55,194 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8164 2025-06-29 03:25:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8b25bda]) (logid:a98fa0bc) No inactive management server node found 2025-06-29 03:25:56,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8b25bda]) (logid:a98fa0bc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:25:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba79bfff]) (logid:cd8b0f36) Enabled non-edge zones available for scan: 1 2025-06-29 03:25:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ba79bfff]) (logid:cd8b0f36) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:25:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba79bfff]) (logid:cd8b0f36) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:25:56,896 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6fe8c7f8]) (logid:e2857948) Enabled non-edge zones available for scan: 1 2025-06-29 03:25:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6fe8c7f8]) (logid:e2857948) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:25:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9d6e42e]) (logid:afb95109) No inactive management server node found 2025-06-29 03:25:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9d6e42e]) (logid:afb95109) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:25:57,953 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-d4249064]) (logid:c81ab72b) AutoScaling Monitor is running... 2025-06-29 03:25:58,081 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-46896288]) (logid:ae0fb578) Starting to remove all volume_stats rows older than [Sat Jun 28 15:25:58 CEST 2025]. 2025-06-29 03:25:58,081 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-46896288]) (logid:ae0fb578) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:25:58 CEST 2025]. 2025-06-29 03:25:58,312 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-77b872ce]) (logid:ec5403e6) HA health check task is running... 2025-06-29 03:25:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03f54afd]) (logid:b7c9eb9e) No inactive management server node found 2025-06-29 03:25:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03f54afd]) (logid:b7c9eb9e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecc79550]) (logid:51f9c03e) No inactive management server node found 2025-06-29 03:26:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecc79550]) (logid:51f9c03e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:02,314 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-506e4e94]) (logid:2a2ddb2a) HA health check task is running... 2025-06-29 03:26:02,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79ac9fb8]) (logid:ddc57b87) No inactive management server node found 2025-06-29 03:26:02,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79ac9fb8]) (logid:ddc57b87) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:03,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99f1fa2b]) (logid:cd6df459) No inactive management server node found 2025-06-29 03:26:03,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99f1fa2b]) (logid:cd6df459) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:05,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8165: Seq 7-8165: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:26:05,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8165 2025-06-29 03:26:05,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46dbccc4]) (logid:a5be12b5) No inactive management server node found 2025-06-29 03:26:05,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46dbccc4]) (logid:a5be12b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:06,153 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:26:06,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:26:06,159 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) Seq 2-3048092522799696775: Routed from 90520738888109 2025-06-29 03:26:06,211 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) Seq 1-3048092522799696775: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:26:06,261 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:26:06,262 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) Seq 2-1612007191621930062: Routed from 90520738888109 2025-06-29 03:26:06,315 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2d7606bf]) (logid:d3d91081) Seq 2-1612007191621930062: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:26:06,315 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-675c8e97]) (logid:e52e30f1) HA health check task is running... 2025-06-29 03:26:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98088a14]) (logid:50380359) No inactive management server node found 2025-06-29 03:26:06,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98088a14]) (logid:50380359) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ad74c56]) (logid:ab9cdb25) No inactive management server node found 2025-06-29 03:26:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ad74c56]) (logid:ab9cdb25) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:09,141 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) StorageCollector is running... 2025-06-29 03:26:09,146 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:26:09,146 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:26:09,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:26:09,147 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 2-6717118844223095501: Routed from 90520738888109 2025-06-29 03:26:09,204 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 4-6717118844223095501: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:26:09,209 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:26:09,209 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:26:09,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:26:09,210 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 2-3048092522799696776: Routed from 90520738888109 2025-06-29 03:26:09,258 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 1-3048092522799696776: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:26:09,280 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:26:09,280 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:26:09,280 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:26:09,282 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 2-3048092522799696777: Routed from 90520738888109 2025-06-29 03:26:09,374 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 1-3048092522799696777: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:26:09,385 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:26:09,385 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:26:09,385 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:26:09,386 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 2-3048092522799696778: Routed from 90520738888109 2025-06-29 03:26:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-149a6520]) (logid:aedab97c) No inactive management server node found 2025-06-29 03:26:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-149a6520]) (logid:aedab97c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:10,316 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6e337c11]) (logid:1c706b3b) HA health check task is running... 2025-06-29 03:26:10,477 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 1-3048092522799696778: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:26:10,479 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:26:10,480 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:26:10,480 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:26:10,481 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 2-1612007191621930063: Routed from 90520738888109 2025-06-29 03:26:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f000da5]) (logid:28a6ca59) Found 0 routers to update status. 2025-06-29 03:26:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f000da5]) (logid:28a6ca59) Found 0 VPC's to update Redundant State. 2025-06-29 03:26:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9f000da5]) (logid:28a6ca59) Found 0 networks to update RvR status. 2025-06-29 03:26:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55c92abf]) (logid:a682581f) No inactive management server node found 2025-06-29 03:26:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55c92abf]) (logid:a682581f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:26:11,500 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-51435b2f]) (logid:e839fe5a) Seq 2-1612007191621930063: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:26:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be4a8389]) (logid:67c823f3) No inactive management server node found 2025-06-29 03:26:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be4a8389]) (logid:67c823f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:26:14,145 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-35b556a3]) (logid:9579432b) Starting to remove all vm_stats rows older than [Sat Jun 28 15:26:14 CEST 2025]. 2025-06-29 03:26:14,170 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-35b556a3]) (logid:9579432b) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:26:14 CEST 2025]. 2025-06-29 03:26:14,317 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ad53b96d]) (logid:295d7022) HA health check task is running... 2025-06-29 03:26:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5394049b]) (logid:ff9487a1) No inactive management server node found 2025-06-29 03:26:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5394049b]) (logid:ff9487a1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:15,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8166: Seq 7-8166: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:26:15,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8166 2025-06-29 03:26:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fab35cf8]) (logid:4499de8c) No inactive management server node found 2025-06-29 03:26:15,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fab35cf8]) (logid:4499de8c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-227254fd]) (logid:4254e334) No inactive management server node found 2025-06-29 03:26:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-227254fd]) (logid:4254e334) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:18,319 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-531e7c8b]) (logid:417ff2f1) HA health check task is running... 2025-06-29 03:26:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39ca6dd0]) (logid:125b9980) No inactive management server node found 2025-06-29 03:26:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39ca6dd0]) (logid:125b9980) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:20,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8167: Seq 7-8167: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:26:20,197 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8167 2025-06-29 03:26:20,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fb95b0f]) (logid:4de7a8b1) No inactive management server node found 2025-06-29 03:26:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fb95b0f]) (logid:4de7a8b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7624a531]) (logid:b4cd6879) No inactive management server node found 2025-06-29 03:26:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7624a531]) (logid:b4cd6879) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:22,320 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8b6183a0]) (logid:a0a074d8) HA health check task is running... 2025-06-29 03:26:23,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-708efc3d]) (logid:8b1b9a6a) No inactive management server node found 2025-06-29 03:26:23,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-708efc3d]) (logid:8b1b9a6a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-207cb064]) (logid:09d611f6) No inactive management server node found 2025-06-29 03:26:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-207cb064]) (logid:09d611f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:26,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af2398a7]) (logid:be8a541a) No inactive management server node found 2025-06-29 03:26:26,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af2398a7]) (logid:be8a541a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:26,322 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4bd5a289]) (logid:86de302a) HA health check task is running... 2025-06-29 03:26:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1260530e]) (logid:f92a8ae2) Enabled non-edge zones available for scan: 1 2025-06-29 03:26:26,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1260530e]) (logid:f92a8ae2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:26:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1260530e]) (logid:f92a8ae2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:26:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d89a2fe3]) (logid:94e38b96) Enabled non-edge zones available for scan: 1 2025-06-29 03:26:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d89a2fe3]) (logid:94e38b96) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:26:27,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb6a2f12]) (logid:a5eb723d) No inactive management server node found 2025-06-29 03:26:27,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb6a2f12]) (logid:a5eb723d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cf9da5a]) (logid:0ca853bb) No inactive management server node found 2025-06-29 03:26:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cf9da5a]) (logid:0ca853bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:30,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8168: Seq 7-8168: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:26:30,195 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8168 2025-06-29 03:26:30,323 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f0663853]) (logid:af183b4c) HA health check task is running... 2025-06-29 03:26:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93e94d0f]) (logid:0dd5abd4) No inactive management server node found 2025-06-29 03:26:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93e94d0f]) (logid:0dd5abd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:30,986 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:26:30,986 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:26:30,987 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) Seq 2-3048092522799696779: Routed from 90520738888109 2025-06-29 03:26:30,990 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) Seq 1-3048092522799696779: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:26:30,990 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:26:30,991 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) Seq 2-1612007191621930064: Routed from 90520738888109 2025-06-29 03:26:30,994 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a71b7345]) (logid:3acdb303) Seq 2-1612007191621930064: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:26:32,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dae77d8]) (logid:e0a993e5) No inactive management server node found 2025-06-29 03:26:32,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dae77d8]) (logid:e0a993e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:33,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24a5ca8e]) (logid:b1e36987) No inactive management server node found 2025-06-29 03:26:33,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24a5ca8e]) (logid:b1e36987) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:26:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:26:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:26:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:26:34,324 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-82dac2fa]) (logid:65cfe9c1) HA health check task is running... 2025-06-29 03:26:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cfe2383]) (logid:85fac95c) No inactive management server node found 2025-06-29 03:26:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cfe2383]) (logid:85fac95c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:36,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb30c7d]) (logid:8635191b) No inactive management server node found 2025-06-29 03:26:36,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb30c7d]) (logid:8635191b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:38,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-def26d06]) (logid:68bafd36) No inactive management server node found 2025-06-29 03:26:38,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-def26d06]) (logid:68bafd36) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:38,326 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8fe3bb4a]) (logid:f10b61dc) HA health check task is running... 2025-06-29 03:26:39,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a7e8ddf]) (logid:2a116b5a) No inactive management server node found 2025-06-29 03:26:39,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a7e8ddf]) (logid:2a116b5a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:40,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8169: Seq 7-8169: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:26:40,203 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8169 2025-06-29 03:26:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1a2fa61a]) (logid:2ec6ee11) Found 0 routers to update status. 2025-06-29 03:26:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1a2fa61a]) (logid:2ec6ee11) Found 0 VPC's to update Redundant State. 2025-06-29 03:26:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1a2fa61a]) (logid:2ec6ee11) Found 0 networks to update RvR status. 2025-06-29 03:26:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-6908b86d]) (logid:cb270a5a) DbCollector is running... 2025-06-29 03:26:41,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a8967a9]) (logid:78c83928) No inactive management server node found 2025-06-29 03:26:41,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a8967a9]) (logid:78c83928) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) ManagementServerCollector is running... 2025-06-29 03:26:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) DB properties were already loaded 2025-06-29 03:26:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:26:42,168 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Execution of process [4063759] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:26:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Exit value of process [4063759] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:26:42,169 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Process [4063759] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:26:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) usage install: null 2025-06-29 03:26:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Sessions active: 0 2025-06-29 03:26:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:26:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063762] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:26:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:26:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063767] for command [/bin/bash -c uname -r ]. 2025-06-29 03:26:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:26:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063768] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:26:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) system memory from /proc: 4887515136 2025-06-29 03:26:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:26:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063773] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:26:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) free memory from /proc: 492924928 2025-06-29 03:26:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:26:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063778] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:26:42,204 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) used memory from /proc: 769160 2025-06-29 03:26:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:26:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063779] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:26:42,210 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:26:42,222 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063780] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:26:42,223 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:26:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063781] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:26:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:26:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063788] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:26:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:26:42,240 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Successfully executed process [4063789] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:26:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail buffersdirect.count, 120 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail buffersdirect.used, 658750 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail buffersmapped.count, 7 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail buffersmapped.used, 798602 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail gcPS-Scavenge.count, 519 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail gcPS-Scavenge.time, 3767 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail jvmuptime, 44342320 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memoryheap.committed, 358088704 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memoryheap.init, 77594624 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memoryheap.usage, 0.10285303901100316 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorynon-heap.max, -1 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorynon-heap.usage, -2.20719312E8 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorynon-heap.used, 220719312 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765164706078379 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.PS-Eden-Space.usage, 0.016038958424855147 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.PS-Old-Gen.usage, 0.1267383789151896 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorypools.PS-Survivor-Space.usage, 0.30599085489908856 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorytotal.committed, 583204864 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorytotal.init, 85262336 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorytotal.max, 1908932607 2025-06-29 03:26:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail memorytotal.used, 417126120 2025-06-29 03:26:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail threadsdeadlocks, [] 2025-06-29 03:26:42,244 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail threadsnew.count, 0 2025-06-29 03:26:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:26:42,248 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-eae12b99]) (logid:4ed9fa08) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:26:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":162481056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44342246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":416326960,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196361912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492924928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.1,0.09],"systemCyclesUsage":[66376093,53815652,4588428183],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:26:42,248 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1477, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:26:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":162481056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44342246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":416326960,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196361912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492924928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.1,0.09],"systemCyclesUsage":[66376093,53815652,4588428183],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:26:42,248 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:26:42,252 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:26:42,252 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1477, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:26:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":162481056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44342246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":416326960,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196361912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492924928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.1,0.09],"systemCyclesUsage":[66376093,53815652,4588428183],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:26:42,252 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-331:[ctx-e9b33358]) (logid:d77cd443) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:26:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":162481056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44342246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":416326960,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":196361912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":492924928,"systemMemoryUsed":769160,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.1,0.09],"systemCyclesUsage":[66376093,53815652,4588428183],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:26:42,327 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4e547e55]) (logid:494e247e) HA health check task is running... 2025-06-29 03:26:42,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ce1ac1b]) (logid:66f3224e) No inactive management server node found 2025-06-29 03:26:42,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ce1ac1b]) (logid:66f3224e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:26:44,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e564fc65]) (logid:eef600e4) No inactive management server node found 2025-06-29 03:26:44,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e564fc65]) (logid:eef600e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:26:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:26:45,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4951d52b]) (logid:3532c43a) No inactive management server node found 2025-06-29 03:26:45,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4951d52b]) (logid:3532c43a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e45de442]) (logid:3dbd64e6) VM scheduler.poll is being called at 2025-06-29T01:27:00+0000 2025-06-29 03:26:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e45de442]) (logid:3dbd64e6) Got 0 scheduled jobs to be executed at 2025-06-29T01:27:00+0000 2025-06-29 03:26:46,052 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e45de442]) (logid:3dbd64e6) Cleaned up 0 VM scheduled job entries 2025-06-29 03:26:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-c298cba4]) (logid:65984d0e) VM Operation Thread Running 2025-06-29 03:26:46,329 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d1206d52]) (logid:383c5ee1) HA health check task is running... 2025-06-29 03:26:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-92546fa6]) (logid:b938a010) ClusterDRS.poll is being called at 2025-06-29T01:27:00+0000 2025-06-29 03:26:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-92546fa6]) (logid:b938a010) Removed 0 old drs migration plans 2025-06-29 03:26:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-fd6008eb]) (logid:98f169d0) Resetting hosts suitable for reconnect 2025-06-29 03:26:47,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-fd6008eb]) (logid:98f169d0) Completed resetting hosts suitable for reconnect 2025-06-29 03:26:47,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-fd6008eb]) (logid:98f169d0) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:26:47,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-fd6008eb]) (logid:98f169d0) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:26:47,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-fd6008eb]) (logid:98f169d0) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:26:47,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-fd6008eb]) (logid:98f169d0) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:26:47,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:26:47,252 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:26:47,254 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:26:47,257 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:26:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e0a05ea]) (logid:c40204f9) No inactive management server node found 2025-06-29 03:26:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e0a05ea]) (logid:c40204f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:48,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5ee6961]) (logid:8f7f84a0) No inactive management server node found 2025-06-29 03:26:48,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5ee6961]) (logid:8f7f84a0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:50,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8171: Seq 7-8171: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:26:50,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8171 2025-06-29 03:26:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-315f3610]) (logid:bce6a938) No inactive management server node found 2025-06-29 03:26:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-315f3610]) (logid:bce6a938) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:50,330 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-56278f2c]) (logid:ed58805a) HA health check task is running... 2025-06-29 03:26:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-553be40c]) (logid:d1b9491b) No inactive management server node found 2025-06-29 03:26:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-553be40c]) (logid:d1b9491b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:53,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f337edc1]) (logid:42f867dc) No inactive management server node found 2025-06-29 03:26:53,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f337edc1]) (logid:42f867dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:54,331 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7e4683da]) (logid:85e61f74) HA health check task is running... 2025-06-29 03:26:54,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8be5c55e]) (logid:c62171e5) No inactive management server node found 2025-06-29 03:26:54,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8be5c55e]) (logid:c62171e5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:56,320 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-398a151d]) (logid:d82fd586) No inactive management server node found 2025-06-29 03:26:56,320 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-398a151d]) (logid:d82fd586) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-60ae7ce9]) (logid:3941842a) Enabled non-edge zones available for scan: 1 2025-06-29 03:26:56,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-60ae7ce9]) (logid:3941842a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:26:56,765 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-60ae7ce9]) (logid:3941842a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:26:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4f87ab6a]) (logid:6560f57e) Enabled non-edge zones available for scan: 1 2025-06-29 03:26:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4f87ab6a]) (logid:6560f57e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:26:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cb747c1]) (logid:3000a5ad) No inactive management server node found 2025-06-29 03:26:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cb747c1]) (logid:3000a5ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:26:57,956 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-b31e21f6]) (logid:4db923c0) AutoScaling Monitor is running... 2025-06-29 03:26:58,083 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-f3853503]) (logid:6823adaf) Starting to remove all volume_stats rows older than [Sat Jun 28 15:26:58 CEST 2025]. 2025-06-29 03:26:58,084 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-f3853503]) (logid:6823adaf) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:26:58 CEST 2025]. 2025-06-29 03:26:58,332 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-689b7e09]) (logid:6ae90ab9) HA health check task is running... 2025-06-29 03:26:59,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fc4deb4]) (logid:f8c23362) No inactive management server node found 2025-06-29 03:26:59,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fc4deb4]) (logid:f8c23362) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:00,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8172: Seq 7-8172: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:00,196 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8172 2025-06-29 03:27:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f88b42d3]) (logid:1a0e2dda) No inactive management server node found 2025-06-29 03:27:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f88b42d3]) (logid:1a0e2dda) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:02,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9b0ff7b]) (logid:ea31be2d) No inactive management server node found 2025-06-29 03:27:02,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9b0ff7b]) (logid:ea31be2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:02,334 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-fdd40660]) (logid:505ca991) HA health check task is running... 2025-06-29 03:27:03,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aad6b655]) (logid:96365955) No inactive management server node found 2025-06-29 03:27:03,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aad6b655]) (logid:96365955) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:05,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e76191f]) (logid:28a11cf0) No inactive management server node found 2025-06-29 03:27:05,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e76191f]) (logid:28a11cf0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:06,335 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ede4c460]) (logid:78c8bb70) HA health check task is running... 2025-06-29 03:27:06,371 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:27:06,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:27:06,376 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) Seq 2-3048092522799696780: Routed from 90520738888109 2025-06-29 03:27:06,429 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) Seq 1-3048092522799696780: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:27:06,461 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:27:06,462 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) Seq 2-1612007191621930065: Routed from 90520738888109 2025-06-29 03:27:06,509 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09bfcbf4]) (logid:55f1964a) Seq 2-1612007191621930065: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:27:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-add2d0f6]) (logid:9fde6100) No inactive management server node found 2025-06-29 03:27:06,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-add2d0f6]) (logid:9fde6100) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:08,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-501c663b]) (logid:ebac53be) No inactive management server node found 2025-06-29 03:27:08,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-501c663b]) (logid:ebac53be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:09,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b035ed8]) (logid:2e0a6a14) No inactive management server node found 2025-06-29 03:27:09,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b035ed8]) (logid:2e0a6a14) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:10,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8173: Seq 7-8173: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:10,185 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8173 2025-06-29 03:27:10,336 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a61e7a3e]) (logid:cdebb7d2) HA health check task is running... 2025-06-29 03:27:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b9de6375]) (logid:24c51b6b) Found 0 routers to update status. 2025-06-29 03:27:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b9de6375]) (logid:24c51b6b) Found 0 VPC's to update Redundant State. 2025-06-29 03:27:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b9de6375]) (logid:24c51b6b) Found 0 networks to update RvR status. 2025-06-29 03:27:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8fd3fbf]) (logid:1cb30ae8) No inactive management server node found 2025-06-29 03:27:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8fd3fbf]) (logid:1cb30ae8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:11,500 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) StorageCollector is running... 2025-06-29 03:27:11,509 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:27:11,509 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:27:11,509 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:27:11,511 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 2-6717118844223095502: Routed from 90520738888109 2025-06-29 03:27:11,566 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 4-6717118844223095502: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:27:11,570 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:27:11,570 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:27:11,570 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:27:11,571 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 2-3048092522799696781: Routed from 90520738888109 2025-06-29 03:27:11,619 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 1-3048092522799696781: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:27:11,642 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:27:11,642 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:27:11,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:27:11,643 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 2-3048092522799696782: Routed from 90520738888109 2025-06-29 03:27:11,730 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 1-3048092522799696782: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:27:11,740 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:27:11,740 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:27:11,740 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:27:11,741 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 2-3048092522799696783: Routed from 90520738888109 2025-06-29 03:27:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c025f42]) (logid:7e68df77) No inactive management server node found 2025-06-29 03:27:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c025f42]) (logid:7e68df77) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:12,832 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 1-3048092522799696783: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:27:12,834 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:27:12,834 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:27:12,834 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:27:12,834 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 2-1612007191621930066: Routed from 90520738888109 2025-06-29 03:27:13,852 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aad97dbb]) (logid:c6116342) Seq 2-1612007191621930066: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:27:14,171 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-1ff14be0]) (logid:80a6f5be) Starting to remove all vm_stats rows older than [Sat Jun 28 15:27:14 CEST 2025]. 2025-06-29 03:27:14,198 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-1ff14be0]) (logid:80a6f5be) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:27:14 CEST 2025]. 2025-06-29 03:27:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3acabb72]) (logid:28f0e01f) No inactive management server node found 2025-06-29 03:27:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3acabb72]) (logid:28f0e01f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:14,338 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-38478eb5]) (logid:0385964e) HA health check task is running... 2025-06-29 03:27:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-096c1dd1]) (logid:162754e9) No inactive management server node found 2025-06-29 03:27:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-096c1dd1]) (logid:162754e9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21fe7d97]) (logid:6bc5e94e) No inactive management server node found 2025-06-29 03:27:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21fe7d97]) (logid:6bc5e94e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:18,339 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b04e8559]) (logid:d42342c8) HA health check task is running... 2025-06-29 03:27:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c58e6b42]) (logid:a58d8a83) No inactive management server node found 2025-06-29 03:27:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c58e6b42]) (logid:a58d8a83) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:20,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8174: Seq 7-8174: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:20,197 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8174 2025-06-29 03:27:20,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51265ea9]) (logid:2ca89bd2) No inactive management server node found 2025-06-29 03:27:20,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51265ea9]) (logid:2ca89bd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8129de50]) (logid:0b0d45ff) No inactive management server node found 2025-06-29 03:27:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8129de50]) (logid:0b0d45ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:22,340 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c504ff2c]) (logid:3a1e6632) HA health check task is running... 2025-06-29 03:27:23,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7960996b]) (logid:13c2d608) No inactive management server node found 2025-06-29 03:27:23,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7960996b]) (logid:13c2d608) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51e752cb]) (logid:dd68185d) No inactive management server node found 2025-06-29 03:27:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51e752cb]) (logid:dd68185d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:25,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8175: Seq 7-8175: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:25,197 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8175 2025-06-29 03:27:26,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8974183]) (logid:47ffad44) No inactive management server node found 2025-06-29 03:27:26,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8974183]) (logid:47ffad44) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:26,341 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a8e3c21d]) (logid:77d04382) HA health check task is running... 2025-06-29 03:27:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3d9ad7a1]) (logid:3cf00886) Enabled non-edge zones available for scan: 1 2025-06-29 03:27:26,765 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3d9ad7a1]) (logid:3cf00886) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:27:26,767 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3d9ad7a1]) (logid:3cf00886) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:27:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3a88e57e]) (logid:806ec07d) Enabled non-edge zones available for scan: 1 2025-06-29 03:27:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3a88e57e]) (logid:806ec07d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:27:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48432049]) (logid:2e49085f) No inactive management server node found 2025-06-29 03:27:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48432049]) (logid:2e49085f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:29,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f689794]) (logid:7dee2ca6) No inactive management server node found 2025-06-29 03:27:29,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f689794]) (logid:7dee2ca6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:30,343 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cfb541d0]) (logid:6a3d97fa) HA health check task is running... 2025-06-29 03:27:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5efbd803]) (logid:eae4d799) No inactive management server node found 2025-06-29 03:27:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5efbd803]) (logid:eae4d799) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:30,996 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:27:30,998 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:27:30,999 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) Seq 2-3048092522799696784: Routed from 90520738888109 2025-06-29 03:27:31,002 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) Seq 1-3048092522799696784: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:27:31,004 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:27:31,005 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) Seq 2-1612007191621930067: Routed from 90520738888109 2025-06-29 03:27:31,008 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d580cc86]) (logid:b075f948) Seq 2-1612007191621930067: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:27:32,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40d28e3b]) (logid:1b60e521) No inactive management server node found 2025-06-29 03:27:32,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40d28e3b]) (logid:1b60e521) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:33,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ef59045]) (logid:a138df11) No inactive management server node found 2025-06-29 03:27:33,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ef59045]) (logid:a138df11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:27:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:27:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:27:33,990 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:27:34,344 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-69c94254]) (logid:25423bcd) HA health check task is running... 2025-06-29 03:27:35,177 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8176: Seq 7-8176: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:35,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8176 2025-06-29 03:27:35,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d53dbac]) (logid:77c2e5a7) No inactive management server node found 2025-06-29 03:27:35,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d53dbac]) (logid:77c2e5a7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68a8fecc]) (logid:1b40fe48) No inactive management server node found 2025-06-29 03:27:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68a8fecc]) (logid:1b40fe48) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:38,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7ca169b]) (logid:e6e5b7db) No inactive management server node found 2025-06-29 03:27:38,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7ca169b]) (logid:e6e5b7db) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:27:38,345 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-791f60cd]) (logid:36dbcab4) HA health check task is running... 2025-06-29 03:27:39,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f22c27df]) (logid:7f406fcd) No inactive management server node found 2025-06-29 03:27:39,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f22c27df]) (logid:7f406fcd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2f4c55e7]) (logid:489686b3) Found 0 routers to update status. 2025-06-29 03:27:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2f4c55e7]) (logid:489686b3) Found 0 VPC's to update Redundant State. 2025-06-29 03:27:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2f4c55e7]) (logid:489686b3) Found 0 networks to update RvR status. 2025-06-29 03:27:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-f20b57e6]) (logid:98973c10) DbCollector is running... 2025-06-29 03:27:41,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5ee4ced]) (logid:49e5864b) No inactive management server node found 2025-06-29 03:27:41,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5ee4ced]) (logid:49e5864b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) ManagementServerCollector is running... 2025-06-29 03:27:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) DB properties were already loaded 2025-06-29 03:27:42,154 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:27:42,169 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Execution of process [4063799] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:27:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Exit value of process [4063799] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:27:42,169 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Process [4063799] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:27:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) usage install: null 2025-06-29 03:27:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Sessions active: 0 2025-06-29 03:27:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:27:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063802] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:27:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:27:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063807] for command [/bin/bash -c uname -r ]. 2025-06-29 03:27:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:27:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063808] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:27:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) system memory from /proc: 4887515136 2025-06-29 03:27:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:27:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063813] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:27:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) free memory from /proc: 489811968 2025-06-29 03:27:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:27:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063818] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:27:42,209 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) used memory from /proc: 769284 2025-06-29 03:27:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:27:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063819] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:27:42,217 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:27:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063820] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:27:42,230 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:27:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063821] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:27:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:27:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063828] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:27:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:27:42,250 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Successfully executed process [4063829] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:27:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail buffersdirect.capacity, 660385 2025-06-29 03:27:42,252 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail buffersdirect.count, 122 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail buffersdirect.used, 660385 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail buffersmapped.count, 7 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail buffersmapped.used, 798602 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail gcPS-Scavenge.count, 520 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail gcPS-Scavenge.time, 3771 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail jvmuptime, 44402330 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memoryheap.committed, 358612992 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memoryheap.init, 77594624 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memoryheap.usage, 0.09951036679027697 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorynon-heap.max, -1 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorynon-heap.usage, -2.2072072E8 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorynon-heap.used, 220720720 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765279258864303 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30336983870430234 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.Metaspace.usage, 0.9859177552347891 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.PS-Eden-Space.usage, 0.0053953569651556555 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.PS-Old-Gen.usage, 0.12678987104261547 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorypools.PS-Survivor-Space.usage, 0.361251220703125 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorytotal.committed, 583729152 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorytotal.init, 85262336 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorytotal.max, 1908932607 2025-06-29 03:27:42,253 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail memorytotal.used, 410735984 2025-06-29 03:27:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail threadsdeadlocks, [] 2025-06-29 03:27:42,254 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail threadsnew.count, 0 2025-06-29 03:27:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) not storing detail threadstimed_waiting.count, 86 2025-06-29 03:27:42,257 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-f4852b13]) (logid:96554b51) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:27:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":169424336,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44402247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409930936,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189958584,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":489811968,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.08],"systemCyclesUsage":[66376424,53815958,4588451289],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:27:42,257 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1479, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:27:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":169424336,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44402247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409930936,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189958584,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":489811968,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.08],"systemCyclesUsage":[66376424,53815958,4588451289],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:27:42,257 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:27:42,263 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:27:42,263 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-65bd7c06]) (logid:ee9a95b5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1479, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:27:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":169424336,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44402247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409930936,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189958584,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":489811968,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.08],"systemCyclesUsage":[66376424,53815958,4588451289],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:27:42,263 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-332:[ctx-24a8de81]) (logid:8b155a0e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:27:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":169424336,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44402247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":409930936,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":189958584,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":219,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":489811968,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.08],"systemCyclesUsage":[66376424,53815958,4588451289],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:27:42,347 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a21e4c60]) (logid:a2039420) HA health check task is running... 2025-06-29 03:27:42,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01f8f031]) (logid:137f4c5a) No inactive management server node found 2025-06-29 03:27:42,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01f8f031]) (logid:137f4c5a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bb14aa4]) (logid:5b66ed16) No inactive management server node found 2025-06-29 03:27:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bb14aa4]) (logid:5b66ed16) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:27:45,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8177: Seq 7-8177: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:45,201 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8177 2025-06-29 03:27:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:27:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:27:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f0a7c8c]) (logid:3607bb27) No inactive management server node found 2025-06-29 03:27:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f0a7c8c]) (logid:3607bb27) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-69f45cc2]) (logid:3bc70555) VM scheduler.poll is being called at 2025-06-29T01:28:00+0000 2025-06-29 03:27:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-69f45cc2]) (logid:3bc70555) Got 0 scheduled jobs to be executed at 2025-06-29T01:28:00+0000 2025-06-29 03:27:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-69f45cc2]) (logid:3bc70555) Cleaned up 0 VM scheduled job entries 2025-06-29 03:27:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-083d88ef]) (logid:8e63837b) VM Operation Thread Running 2025-06-29 03:27:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f2ebced4]) (logid:9635600d) ClusterDRS.poll is being called at 2025-06-29T01:28:00+0000 2025-06-29 03:27:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f2ebced4]) (logid:9635600d) Removed 0 old drs migration plans 2025-06-29 03:27:46,348 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b44f5a28]) (logid:8353abbb) HA health check task is running... 2025-06-29 03:27:47,256 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:27:47,257 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:27:47,259 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:27:47,262 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:27:47,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07b4e716]) (logid:f2540b6a) No inactive management server node found 2025-06-29 03:27:47,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07b4e716]) (logid:f2540b6a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4d14615]) (logid:cc09bff3) No inactive management server node found 2025-06-29 03:27:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4d14615]) (logid:cc09bff3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9435386]) (logid:9bff0556) No inactive management server node found 2025-06-29 03:27:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9435386]) (logid:9bff0556) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:50,349 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b1278851]) (logid:5b0e701c) HA health check task is running... 2025-06-29 03:27:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da88bb18]) (logid:6962e276) No inactive management server node found 2025-06-29 03:27:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da88bb18]) (logid:6962e276) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:53,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24921dd8]) (logid:e86c585e) No inactive management server node found 2025-06-29 03:27:53,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24921dd8]) (logid:e86c585e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:27:54,350 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d3152645]) (logid:48abd96b) HA health check task is running... 2025-06-29 03:27:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7f0ca5c]) (logid:0e59c567) No inactive management server node found 2025-06-29 03:27:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7f0ca5c]) (logid:0e59c567) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:55,178 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8179: Seq 7-8179: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:27:55,199 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8179 2025-06-29 03:27:56,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0748b5b6]) (logid:9b65c117) No inactive management server node found 2025-06-29 03:27:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0748b5b6]) (logid:9b65c117) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7b9f60be]) (logid:1fe110d1) Enabled non-edge zones available for scan: 1 2025-06-29 03:27:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-7b9f60be]) (logid:1fe110d1) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:27:56,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7b9f60be]) (logid:1fe110d1) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:27:56,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-875c2c2c]) (logid:06668e35) Enabled non-edge zones available for scan: 1 2025-06-29 03:27:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-875c2c2c]) (logid:06668e35) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:27:57,836 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cec9e8d5]) (logid:46042529) No inactive management server node found 2025-06-29 03:27:57,836 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cec9e8d5]) (logid:46042529) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:27:57,958 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-3ebbcd77]) (logid:051e9876) AutoScaling Monitor is running... 2025-06-29 03:27:58,086 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-8df07c3e]) (logid:9c93677c) Starting to remove all volume_stats rows older than [Sat Jun 28 15:27:58 CEST 2025]. 2025-06-29 03:27:58,086 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-8df07c3e]) (logid:9c93677c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:27:58 CEST 2025]. 2025-06-29 03:27:58,351 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-42a7afde]) (logid:21ec6c7a) HA health check task is running... 2025-06-29 03:27:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-972727b5]) (logid:9105a8bf) No inactive management server node found 2025-06-29 03:27:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-972727b5]) (logid:9105a8bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:00,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb7a4d15]) (logid:dc82a3d6) No inactive management server node found 2025-06-29 03:28:00,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb7a4d15]) (logid:dc82a3d6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:02,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9bec52]) (logid:ab8a7e61) No inactive management server node found 2025-06-29 03:28:02,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9bec52]) (logid:ab8a7e61) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:02,352 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0ba6de3f]) (logid:64c0ebfe) HA health check task is running... 2025-06-29 03:28:03,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a54e45]) (logid:271c869b) No inactive management server node found 2025-06-29 03:28:03,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a54e45]) (logid:271c869b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:05,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8180: Seq 7-8180: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:05,203 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8180 2025-06-29 03:28:05,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae25d6a5]) (logid:e0a496f1) No inactive management server node found 2025-06-29 03:28:05,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae25d6a5]) (logid:e0a496f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:06,353 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-90465148]) (logid:88ac4004) HA health check task is running... 2025-06-29 03:28:06,562 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:28:06,566 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:28:06,568 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) Seq 2-3048092522799696785: Routed from 90520738888109 2025-06-29 03:28:06,623 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) Seq 1-3048092522799696785: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:28:06,667 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:28:06,668 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) Seq 2-1612007191621930068: Routed from 90520738888109 2025-06-29 03:28:06,709 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4a29b520]) (logid:c2ea87c9) Seq 2-1612007191621930068: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:28:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36d57633]) (logid:bf774bab) No inactive management server node found 2025-06-29 03:28:06,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36d57633]) (logid:bf774bab) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef3a0f5e]) (logid:46ddbdd8) No inactive management server node found 2025-06-29 03:28:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef3a0f5e]) (logid:46ddbdd8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:09,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa2370e2]) (logid:9d72b3de) No inactive management server node found 2025-06-29 03:28:09,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa2370e2]) (logid:9d72b3de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:10,355 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-aae62a03]) (logid:7b2e64d3) HA health check task is running... 2025-06-29 03:28:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-40c39634]) (logid:6487f687) Found 0 routers to update status. 2025-06-29 03:28:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-40c39634]) (logid:6487f687) Found 0 VPC's to update Redundant State. 2025-06-29 03:28:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-40c39634]) (logid:6487f687) Found 0 networks to update RvR status. 2025-06-29 03:28:11,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7a9ad45]) (logid:ea468fb2) No inactive management server node found 2025-06-29 03:28:11,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7a9ad45]) (logid:ea468fb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:12,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05e0a4e1]) (logid:e1787110) No inactive management server node found 2025-06-29 03:28:12,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05e0a4e1]) (logid:e1787110) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:13,853 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) StorageCollector is running... 2025-06-29 03:28:13,858 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:28:13,858 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:28:13,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:28:13,859 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-6717118844223095503: Routed from 90520738888109 2025-06-29 03:28:13,936 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 4-6717118844223095503: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:28:13,941 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:28:13,941 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:28:13,941 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:28:13,942 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-3048092522799696786: Routed from 90520738888109 2025-06-29 03:28:13,990 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 1-3048092522799696786: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:28:14,015 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:28:14,016 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:28:14,016 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:28:14,017 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-1612007191621930069: Routed from 90520738888109 2025-06-29 03:28:14,069 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-1612007191621930069: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:28:14,079 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:28:14,079 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:28:14,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:28:14,080 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-3048092522799696787: Routed from 90520738888109 2025-06-29 03:28:14,199 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-f39fe60c]) (logid:dfd7f5e2) Starting to remove all vm_stats rows older than [Sat Jun 28 15:28:14 CEST 2025]. 2025-06-29 03:28:14,226 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-f39fe60c]) (logid:dfd7f5e2) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:28:14 CEST 2025]. 2025-06-29 03:28:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb9c880f]) (logid:b2ab455d) No inactive management server node found 2025-06-29 03:28:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb9c880f]) (logid:b2ab455d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:14,356 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8a3a0051]) (logid:c35c5cba) HA health check task is running... 2025-06-29 03:28:15,132 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 1-3048092522799696787: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:28:15,134 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:28:15,134 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:28:15,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:28:15,135 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-1612007191621930070: Routed from 90520738888109 2025-06-29 03:28:15,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8181: Seq 7-8181: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:15,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8181 2025-06-29 03:28:15,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d65c435a]) (logid:b875e433) No inactive management server node found 2025-06-29 03:28:15,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d65c435a]) (logid:b875e433) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:16,202 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-4270ef89]) (logid:062c298e) Seq 2-1612007191621930070: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:28:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17c8742b]) (logid:0c7a3487) Resetting hosts suitable for reconnect 2025-06-29 03:28:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17c8742b]) (logid:0c7a3487) Completed resetting hosts suitable for reconnect 2025-06-29 03:28:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17c8742b]) (logid:0c7a3487) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:28:17,060 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17c8742b]) (logid:0c7a3487) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:28:17,060 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17c8742b]) (logid:0c7a3487) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:28:17,060 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17c8742b]) (logid:0c7a3487) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:28:17,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c94b2f8]) (logid:978fc41d) No inactive management server node found 2025-06-29 03:28:17,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c94b2f8]) (logid:978fc41d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:18,357 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-df9136fe]) (logid:7b77f02b) HA health check task is running... 2025-06-29 03:28:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c63add9]) (logid:fe095c23) No inactive management server node found 2025-06-29 03:28:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c63add9]) (logid:fe095c23) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:20,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50598162]) (logid:5e987884) No inactive management server node found 2025-06-29 03:28:20,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50598162]) (logid:5e987884) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:21,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9865a2ac]) (logid:b7d8a4a4) No inactive management server node found 2025-06-29 03:28:21,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9865a2ac]) (logid:b7d8a4a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:22,358 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c45c6d8d]) (logid:e51c71e9) HA health check task is running... 2025-06-29 03:28:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-051cc795]) (logid:66eb84fe) No inactive management server node found 2025-06-29 03:28:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-051cc795]) (logid:66eb84fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39bdeb90]) (logid:e6c0480a) No inactive management server node found 2025-06-29 03:28:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39bdeb90]) (logid:e6c0480a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:25,179 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8182: Seq 7-8182: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:25,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8182 2025-06-29 03:28:26,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6461e2d3]) (logid:07f857ad) No inactive management server node found 2025-06-29 03:28:26,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6461e2d3]) (logid:07f857ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:26,360 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2bac1dbf]) (logid:d805ac61) HA health check task is running... 2025-06-29 03:28:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-31fff23a]) (logid:86e15f75) Enabled non-edge zones available for scan: 1 2025-06-29 03:28:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-31fff23a]) (logid:86e15f75) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:28:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-31fff23a]) (logid:86e15f75) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:28:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1c67c35f]) (logid:167ca437) Enabled non-edge zones available for scan: 1 2025-06-29 03:28:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1c67c35f]) (logid:167ca437) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:28:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc05b574]) (logid:0b176ea7) No inactive management server node found 2025-06-29 03:28:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc05b574]) (logid:0b176ea7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfb628bb]) (logid:bf9f5b73) No inactive management server node found 2025-06-29 03:28:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfb628bb]) (logid:bf9f5b73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:30,361 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-91824bcf]) (logid:5c32925a) HA health check task is running... 2025-06-29 03:28:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22b51938]) (logid:1384156c) No inactive management server node found 2025-06-29 03:28:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22b51938]) (logid:1384156c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:31,011 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:28:31,011 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:28:31,012 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) Seq 2-3048092522799696788: Routed from 90520738888109 2025-06-29 03:28:31,016 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) Seq 1-3048092522799696788: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:28:31,017 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:28:31,018 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) Seq 2-1612007191621930071: Routed from 90520738888109 2025-06-29 03:28:31,021 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-1aa2f3dd]) (logid:5c0b6fb7) Seq 2-1612007191621930071: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:28:32,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9401fab]) (logid:1a5d511b) No inactive management server node found 2025-06-29 03:28:32,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9401fab]) (logid:1a5d511b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ff41c30]) (logid:ea562b84) No inactive management server node found 2025-06-29 03:28:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ff41c30]) (logid:ea562b84) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:33,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:28:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:28:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:28:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:28:34,362 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-73944824]) (logid:0ac9326a) HA health check task is running... 2025-06-29 03:28:35,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8183: Seq 7-8183: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:35,203 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8183 2025-06-29 03:28:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e41eed5]) (logid:0abe036d) No inactive management server node found 2025-06-29 03:28:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e41eed5]) (logid:0abe036d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edfcbc5a]) (logid:0233ff56) No inactive management server node found 2025-06-29 03:28:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edfcbc5a]) (logid:0233ff56) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d22838f]) (logid:3b5845d4) No inactive management server node found 2025-06-29 03:28:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d22838f]) (logid:3b5845d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:38,364 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9d9091dd]) (logid:3087b4d5) HA health check task is running... 2025-06-29 03:28:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f1af89f]) (logid:8e00c900) No inactive management server node found 2025-06-29 03:28:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f1af89f]) (logid:8e00c900) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:40,816 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-85b105f3]) (logid:d033b6e9) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:28:40,817 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-85b105f3]) (logid:d033b6e9) Snapshot scheduler is being called at [2025-06-29T01:28:40+0000]. 2025-06-29 03:28:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-85b105f3]) (logid:d033b6e9) There are [0] scheduled snapshots to be executed at [2025-06-29T01:28:40+0000]. 2025-06-29 03:28:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-4692db5d]) (logid:ca92ec20) External devices are not used. Skipping external device usage collection 2025-06-29 03:28:40,946 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-5141ecbb]) (logid:a1f4c567) checking health of usage server 2025-06-29 03:28:40,948 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-5141ecbb]) (logid:a1f4c567) usage server running? false, heartbeat: null 2025-06-29 03:28:40,952 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-5141ecbb]) (logid:a1f4c567) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 03:28:40,954 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-5141ecbb]) (logid:a1f4c567) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 03:28:40,980 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-c46e296e]) (logid:245dfc16) Backup backup.poll is being called at 2025-06-29T01:28:40+0000 2025-06-29 03:28:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-a25424d4]) (logid:f67c48ef) Found 0 running routers. 2025-06-29 03:28:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9b79753e]) (logid:928eff01) Found 0 routers to update status. 2025-06-29 03:28:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9b79753e]) (logid:928eff01) Found 0 VPC's to update Redundant State. 2025-06-29 03:28:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9b79753e]) (logid:928eff01) Found 0 networks to update RvR status. 2025-06-29 03:28:41,018 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Found 1 running routers. 2025-06-29 03:28:41,024 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:28:41,026 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Seq 2-1612007191621930072: Routed from 90520738888109 2025-06-29 03:28:41,026 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Seq 2-1612007191621930072: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:41,054 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-b2daff05]) (logid:78bbb6ff) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 03:28:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-c2b90289]) (logid:11791cbf) DbCollector is running... 2025-06-29 03:28:41,287 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-100cd638]) (logid:70d94ea6) Started resource counters recalculation periodic task. 2025-06-29 03:28:41,291 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-100cd638]) (logid:70d94ea6) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:28:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c28d7ef]) (logid:a3c65178) No inactive management server node found 2025-06-29 03:28:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c28d7ef]) (logid:a3c65178) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:28:41,652 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-100cd638]) (logid:70d94ea6) Finished resource counters recalculation periodic task. 2025-06-29 03:28:41,690 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 2-1612007191621930072: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:41,690 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Seq 2-1612007191621930072: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:28:41,690 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 03:28:41,693 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 03:28:41,734 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 03:28:41,736 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Seq 2-1612007191621930073: Routed from 90520738888109 2025-06-29 03:28:41,736 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Seq 2-1612007191621930073: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) ManagementServerCollector is running... 2025-06-29 03:28:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) DB properties were already loaded 2025-06-29 03:28:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:28:42,168 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Execution of process [4063845] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:28:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Exit value of process [4063845] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:28:42,169 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Process [4063845] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:28:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) usage install: null 2025-06-29 03:28:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Sessions active: 0 2025-06-29 03:28:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:28:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063848] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:28:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:28:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063853] for command [/bin/bash -c uname -r ]. 2025-06-29 03:28:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:28:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063854] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:28:42,187 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) system memory from /proc: 4887515136 2025-06-29 03:28:42,187 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:28:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063859] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:28:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) free memory from /proc: 493088768 2025-06-29 03:28:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:28:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063864] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:28:42,209 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) used memory from /proc: 769284 2025-06-29 03:28:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:28:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063865] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:28:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:28:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063866] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:28:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:28:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063867] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:28:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:28:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063874] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:28:42,247 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:28:42,253 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Successfully executed process [4063875] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail buffersdirect.count, 120 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail buffersdirect.used, 658750 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail buffersmapped.count, 7 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail buffersmapped.used, 798602 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail gcPS-Scavenge.count, 521 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail gcPS-Scavenge.time, 3776 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail jvmuptime, 44462333 2025-06-29 03:28:42,255 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memoryheap.committed, 358088704 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memoryheap.init, 77594624 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memoryheap.usage, 0.10368031179862375 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorynon-heap.max, -1 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorynon-heap.usage, -2.20722648E8 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorynon-heap.used, 220722648 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765279258864303 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.3033823357883161 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.Metaspace.usage, 0.9859203972159342 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.PS-Eden-Space.usage, 0.014739932348225295 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.PS-Old-Gen.usage, 0.1268127564325825 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorypools.PS-Survivor-Space.usage, 0.5135869565217391 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorytotal.committed, 583204864 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorytotal.init, 85262336 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorytotal.max, 1908932607 2025-06-29 03:28:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail memorytotal.used, 418692776 2025-06-29 03:28:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail threadsdeadlocks, [] 2025-06-29 03:28:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail threadsnew.count, 0 2025-06-29 03:28:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:28:42,261 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-d2726b47]) (logid:66b9d9bd) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:28:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":160921600,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44462246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":417889752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197918728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493088768,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.06,0.08],"systemCyclesUsage":[66376773,53816280,4588474357],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:28:42,261 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1481, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:28:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":160921600,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44462246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":417889752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197918728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493088768,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.06,0.08],"systemCyclesUsage":[66376773,53816280,4588474357],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:28:42,261 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:28:42,266 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:28:42,266 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1481, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:28:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":160921600,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44462246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":417889752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197918728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493088768,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.06,0.08],"systemCyclesUsage":[66376773,53816280,4588474357],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:28:42,266 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-333:[ctx-0baf1b99]) (logid:f541d320) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:28:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":160921600,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44462246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":417889752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":197918728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493088768,"systemMemoryUsed":769284,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.06,0.08],"systemCyclesUsage":[66376773,53816280,4588474357],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:28:42,365 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4fecb57e]) (logid:4acfda78) HA health check task is running... 2025-06-29 03:28:42,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35135874]) (logid:97511022) No inactive management server node found 2025-06-29 03:28:42,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35135874]) (logid:97511022) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:43,149 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 2-1612007191621930073: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.82e0fe56-c58d-4eff-ae1a-f822010a80d6","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:43,149 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-b247726b]) (logid:8fdc2f31) Seq 2-1612007191621930073: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 03:28:43,151 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 03:28:43,155 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:28:43,156 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Seq 2-1612007191621930074: Routed from 90520738888109 2025-06-29 03:28:43,157 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Seq 2-1612007191621930074: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:43,633 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 2-1612007191621930074: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "9.615182876586914", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "18.865346908569336", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "27.321338653564453", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "35.89653968811035", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751160421565", "lastRunDuration": "43.250083923339844", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751160421630", "lastRunDuration": "8068.703174591064", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751160429740", "lastRunDuration": "58.17842483520508", "message": "Memory Usage within limits with current at 50.886%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751160429798", "lastRunDuration": "3224.2391109466553", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751160433071", "lastRunDuration": "41.556596755981445", "message": "Sufficient free space is 2571.46875 MB"}, "lastRun": {"start": "2025-06-29 01:27:01.431283", "end": "2025-06-29 01:27:13.113642", "duration": "11.682359218597412"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751160001216", "lastRunDuration": "47.1339225769043", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751160001263", "lastRunDuration": "54.06928062438965", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751160001444", "lastRunDuration": "42.38390922546387", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751160001596", "lastRunDuration": "42.78373718261719", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:20:01.213608", "end": "2025-06-29 01:20:01.683779", "duration": "0.4701709747314453"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "9.615182876586914", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "18.865346908569336", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "27.321338653564453", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "35.89653968811035", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751160421565", "lastRunDuration": "43.250083923339844", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751160421630", "lastRunDuration": "8068.703174591064", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751160429740", "lastRunDuration": "58.17842483520508", "message": "Memory Usage within limits with current at 50.886%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751160429798", "lastRunDuration": "3224.2391109466553", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751160433071", "lastRunDuration": "41.556596755981445", "message": "Sufficient free space is 2571.46875 MB"}, "lastRun": {"start": "2025-06-29 01:27:01.431283", "end": "2025-06-29 01:27:13.113642", "duration": "11.682359218597412"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751160001216", "lastRunDuration": "47.1339225769043", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751160001263", "lastRunDuration": "54.06928062438965", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751160001444", "lastRunDuration": "42.38390922546387", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751160001596", "lastRunDuration": "42.78373718261719", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:20:01.213608", "end": "2025-06-29 01:20:01.683779", "duration": "0.4701709747314453"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:43,633 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Seq 2-1612007191621930074: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:28:43,633 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "9.615182876586914", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "18.865346908569336", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "27.321338653564453", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "35.89653968811035", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751160421565", "lastRunDuration": "43.250083923339844", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751160421630", "lastRunDuration": "8068.703174591064", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751160429740", "lastRunDuration": "58.17842483520508", "message": "Memory Usage within limits with current at 50.886%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751160429798", "lastRunDuration": "3224.2391109466553", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751160433071", "lastRunDuration": "41.556596755981445", "message": "Sufficient free space is 2571.46875 MB"}, "lastRun": {"start": "2025-06-29 01:27:01.431283", "end": "2025-06-29 01:27:13.113642", "duration": "11.682359218597412"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751160001216", "lastRunDuration": "47.1339225769043", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751160001263", "lastRunDuration": "54.06928062438965", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751160001444", "lastRunDuration": "42.38390922546387", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751160001596", "lastRunDuration": "42.78373718261719", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:20:01.213608", "end": "2025-06-29 01:20:01.683779", "duration": "0.4701709747314453"}}} 2025-06-29 03:28:43,667 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "9.615182876586914", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "18.865346908569336", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "27.321338653564453", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751160421433", "lastRunDuration": "35.89653968811035", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751160421565", "lastRunDuration": "43.250083923339844", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751160421630", "lastRunDuration": "8068.703174591064", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751160429740", "lastRunDuration": "58.17842483520508", "message": "Memory Usage within limits with current at 50.886%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751160429798", "lastRunDuration": "3224.2391109466553", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751160433071", "lastRunDuration": "41.556596755981445", "message": "Sufficient free space is 2571.46875 MB"}, "lastRun": {"start": "2025-06-29 01:27:01.431283", "end": "2025-06-29 01:27:13.113642", "duration": "11.682359218597412"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751160001216", "lastRunDuration": "47.1339225769043", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751160001263", "lastRunDuration": "54.06928062438965", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751160001444", "lastRunDuration": "42.38390922546387", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751160001596", "lastRunDuration": "42.78373718261719", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:20:01.213608", "end": "2025-06-29 01:20:01.683779", "duration": "0.4701709747314453"}}} 2025-06-29 03:28:43,668 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found check types executed on VR basic, start: 2025-06-29 01:27:01.431283, end: 2025-06-29 01:27:13.113642, duration: 11.682359218597412 2025-06-29 03:28:43,677 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@27dbab16- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 03:27:01 CEST 2025, details: service is running which took running duration (ms) 9.615182876586914 2025-06-29 03:28:43,686 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@dea8b44- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 03:27:01 CEST 2025, details: service is running which took running duration (ms) 18.865346908569336 2025-06-29 03:28:43,697 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@47a5650- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 03:27:01 CEST 2025, details: service is running which took running duration (ms) 27.321338653564453 2025-06-29 03:28:43,707 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4ba81a0e- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 03:27:01 CEST 2025, details: service is running which took running duration (ms) 35.89653968811035 2025-06-29 03:28:43,718 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@31519b76- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 03:27:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 43.250083923339844 2025-06-29 03:28:43,728 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@683896b3- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 03:27:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8068.703174591064 2025-06-29 03:28:43,738 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@3ea68532- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 03:27:09 CEST 2025, details: Memory Usage within limits with current at 50.886% which took running duration (ms) 58.17842483520508 2025-06-29 03:28:43,751 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@35559082- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 03:27:09 CEST 2025, details: CPU Usage within limits with current at 0.6% which took running duration (ms) 3224.2391109466553 2025-06-29 03:28:43,762 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@8e22775- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 03:27:13 CEST 2025, details: Sufficient free space is 2571.46875 MB which took running duration (ms) 41.556596755981445 2025-06-29 03:28:43,762 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found check types executed on VR advanced, start: 2025-06-29 01:20:01.213608, end: 2025-06-29 01:20:01.683779, duration: 0.4701709747314453 2025-06-29 03:28:43,773 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@24c3b5d4- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 03:20:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 47.1339225769043 2025-06-29 03:28:43,783 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2e462b6b- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 03:20:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 54.06928062438965 2025-06-29 03:28:43,793 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@378e7fa1- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 03:20:01 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 42.38390922546387 2025-06-29 03:28:43,804 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2dff6b16]) (logid:8b353770) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2ece2faa- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 03:20:01 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 42.78373718261719 2025-06-29 03:28:44,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80b61589]) (logid:fed7dd00) No inactive management server node found 2025-06-29 03:28:44,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80b61589]) (logid:fed7dd00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:45,180 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8184: Seq 7-8184: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:45,193 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8184 2025-06-29 03:28:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:28:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:28:45,836 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d400df9]) (logid:dfc34b5d) No inactive management server node found 2025-06-29 03:28:45,837 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d400df9]) (logid:dfc34b5d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-08990618]) (logid:41d60142) VM scheduler.poll is being called at 2025-06-29T01:29:00+0000 2025-06-29 03:28:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-08990618]) (logid:41d60142) Got 0 scheduled jobs to be executed at 2025-06-29T01:29:00+0000 2025-06-29 03:28:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-08990618]) (logid:41d60142) Cleaned up 0 VM scheduled job entries 2025-06-29 03:28:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-8b57b81f]) (logid:50d60264) VM Operation Thread Running 2025-06-29 03:28:46,191 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-2:[ctx-de95d470]) (logid:211bd064) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:28:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-93801d32]) (logid:45b5061e) ClusterDRS.poll is being called at 2025-06-29T01:29:00+0000 2025-06-29 03:28:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-93801d32]) (logid:45b5061e) Removed 0 old drs migration plans 2025-06-29 03:28:46,366 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7f727543]) (logid:09355ba9) HA health check task is running... 2025-06-29 03:28:47,260 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:28:47,262 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:28:47,264 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:28:47,267 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:28:47,327 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9de32fa9]) (logid:b1816b0b) No inactive management server node found 2025-06-29 03:28:47,327 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9de32fa9]) (logid:b1816b0b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e707e415]) (logid:379178e5) No inactive management server node found 2025-06-29 03:28:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e707e415]) (logid:379178e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:50,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8186: Seq 7-8186: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:28:50,202 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8186 2025-06-29 03:28:50,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59392bd0]) (logid:adbb1aa8) No inactive management server node found 2025-06-29 03:28:50,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59392bd0]) (logid:adbb1aa8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:50,367 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3a32c1d8]) (logid:af590016) HA health check task is running... 2025-06-29 03:28:51,031 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-666cfb02]) (logid:ea51b36f) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:28:51,040 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-666cfb02]) (logid:ea51b36f) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:28:51,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b586eae]) (logid:42bae653) No inactive management server node found 2025-06-29 03:28:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b586eae]) (logid:42bae653) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:53,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfb16d92]) (logid:8d77f378) No inactive management server node found 2025-06-29 03:28:53,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfb16d92]) (logid:8d77f378) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:54,368 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d4c3791d]) (logid:17862b43) HA health check task is running... 2025-06-29 03:28:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb620d13]) (logid:bc333036) No inactive management server node found 2025-06-29 03:28:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb620d13]) (logid:bc333036) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:56,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:28:56,212 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Seq 2-3048092522799696789: Routed from 90520738888109 2025-06-29 03:28:56,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a27924c]) (logid:69c7c515) No inactive management server node found 2025-06-29 03:28:56,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a27924c]) (logid:69c7c515) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-94eb4826]) (logid:238ae85b) Enabled non-edge zones available for scan: 1 2025-06-29 03:28:56,764 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-94eb4826]) (logid:238ae85b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:28:56,765 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-94eb4826]) (logid:238ae85b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:28:56,896 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9a04944f]) (logid:32099bb9) Enabled non-edge zones available for scan: 1 2025-06-29 03:28:56,903 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9a04944f]) (logid:32099bb9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:28:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8042199e]) (logid:491f2bb7) No inactive management server node found 2025-06-29 03:28:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8042199e]) (logid:491f2bb7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:28:57,960 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-cbb71579]) (logid:30099476) AutoScaling Monitor is running... 2025-06-29 03:28:58,089 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-64b046a7]) (logid:90bd0d3e) Starting to remove all volume_stats rows older than [Sat Jun 28 15:28:58 CEST 2025]. 2025-06-29 03:28:58,089 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-64b046a7]) (logid:90bd0d3e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:28:58 CEST 2025]. 2025-06-29 03:28:58,370 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-408e5105]) (logid:3d613a8b) HA health check task is running... 2025-06-29 03:28:59,278 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Seq 1-3048092522799696789: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:28:59,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 03:28:59,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:28:59,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Seq 2-1612007191621930075: Routed from 90520738888109 2025-06-29 03:28:59,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-184309e6]) (logid:c7f3b3e6) No inactive management server node found 2025-06-29 03:28:59,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-184309e6]) (logid:c7f3b3e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:00,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8187: Seq 7-8187: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:29:00,205 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8187 2025-06-29 03:29:00,830 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4685de07]) (logid:2003444a) No inactive management server node found 2025-06-29 03:29:00,830 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4685de07]) (logid:2003444a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:02,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-694f849d]) (logid:cbc7c282) No inactive management server node found 2025-06-29 03:29:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-694f849d]) (logid:cbc7c282) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:02,371 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d134ac24]) (logid:6faab92d) HA health check task is running... 2025-06-29 03:29:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c25dccba]) (logid:16a1a2a2) No inactive management server node found 2025-06-29 03:29:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c25dccba]) (logid:16a1a2a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:04,452 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Seq 2-1612007191621930075: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:29:04,452 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1522e544]) (logid:936421e3) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: 2025-06-29 03:29:05,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-056430de]) (logid:1f6e2c57) No inactive management server node found 2025-06-29 03:29:05,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-056430de]) (logid:1f6e2c57) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:29:06,373 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-41b2b556]) (logid:ad90800e) HA health check task is running... 2025-06-29 03:29:06,764 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:29:06,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:29:06,768 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) Seq 2-3048092522799696790: Routed from 90520738888109 2025-06-29 03:29:06,779 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) Seq 1-3048092522799696790: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:29:06,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-167df7e2]) (logid:13adc753) No inactive management server node found 2025-06-29 03:29:06,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-167df7e2]) (logid:13adc753) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:06,826 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:29:06,827 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) Seq 2-1612007191621930076: Routed from 90520738888109 2025-06-29 03:29:06,869 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d631b473]) (logid:acb4b9d2) Seq 2-1612007191621930076: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:29:08,326 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83439907]) (logid:2debefe7) No inactive management server node found 2025-06-29 03:29:08,326 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83439907]) (logid:2debefe7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:09,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06c32860]) (logid:5a002a8d) No inactive management server node found 2025-06-29 03:29:09,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06c32860]) (logid:5a002a8d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:10,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8188: Seq 7-8188: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:29:10,202 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8188 2025-06-29 03:29:10,374 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9251a0fd]) (logid:c24c8559) HA health check task is running... 2025-06-29 03:29:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f1d49fd4]) (logid:8a25635a) Found 0 routers to update status. 2025-06-29 03:29:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f1d49fd4]) (logid:8a25635a) Found 0 VPC's to update Redundant State. 2025-06-29 03:29:11,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f1d49fd4]) (logid:8a25635a) Found 0 networks to update RvR status. 2025-06-29 03:29:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Running Capacity Checker ... 2025-06-29 03:29:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) recalculating system capacity 2025-06-29 03:29:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Executing cpu/ram capacity update 2025-06-29 03:29:11,073 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:11,074 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:11,084 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:11,090 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:29:11,091 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:29:11,091 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:29:11,114 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:11,115 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:11,119 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:11,121 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:29:11,121 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:29:11,121 DEBUG [c.c.c.CapacityManagerImpl] (pool-324-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:29:11,130 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Done executing cpu/ram capacity update 2025-06-29 03:29:11,130 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Executing storage capacity update 2025-06-29 03:29:11,138 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:29:11,138 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:29:11,139 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:29:11,142 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:29:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:29:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:29:11,148 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:29:11,151 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:29:11,161 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:29:11,161 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:29:11,161 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:29:11,165 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:29:11,171 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:29:11,171 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:29:11,171 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:29:11,175 DEBUG [c.c.s.StorageManagerImpl] (pool-325-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:29:11,175 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Done executing storage capacity update 2025-06-29 03:29:11,176 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Executing capacity updates for public ip and Vlans 2025-06-29 03:29:11,191 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Done capacity updates for public ip and Vlans 2025-06-29 03:29:11,191 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Executing capacity updates for private ip 2025-06-29 03:29:11,194 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Done executing capacity updates for private ip 2025-06-29 03:29:11,194 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Done recalculating system capacity 2025-06-29 03:29:11,223 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-34d20228]) (logid:ce48bcd3) Done running Capacity Checker ... 2025-06-29 03:29:11,261 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-55b449bd]) (logid:c4d7123e) Checking if any host reservation can be released ... 2025-06-29 03:29:11,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-55b449bd]) (logid:c4d7123e) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:11,271 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-55b449bd]) (logid:c4d7123e) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:11,272 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-55b449bd]) (logid:c4d7123e) Done running HostReservationReleaseChecker ... 2025-06-29 03:29:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1837d463]) (logid:55eca723) No inactive management server node found 2025-06-29 03:29:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1837d463]) (logid:55eca723) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:12,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-106c3145]) (logid:64c88fd4) No inactive management server node found 2025-06-29 03:29:12,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-106c3145]) (logid:64c88fd4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:14,227 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-4a9b3a94]) (logid:c1cfe7ae) Starting to remove all vm_stats rows older than [Sat Jun 28 15:29:14 CEST 2025]. 2025-06-29 03:29:14,271 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-4a9b3a94]) (logid:c1cfe7ae) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:29:14 CEST 2025]. 2025-06-29 03:29:14,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ca11528]) (logid:762a7957) No inactive management server node found 2025-06-29 03:29:14,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ca11528]) (logid:762a7957) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:29:14,375 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-71b6281a]) (logid:55aff16c) HA health check task is running... 2025-06-29 03:29:15,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45e3ae55]) (logid:8cf1187a) No inactive management server node found 2025-06-29 03:29:15,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45e3ae55]) (logid:8cf1187a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:16,203 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) StorageCollector is running... 2025-06-29 03:29:16,211 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:29:16,211 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:29:16,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:29:16,216 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-6717118844223095504: Routed from 90520738888109 2025-06-29 03:29:16,271 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 4-6717118844223095504: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:29:16,276 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:29:16,276 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:29:16,276 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:29:16,277 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-1612007191621930077: Routed from 90520738888109 2025-06-29 03:29:16,290 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-1612007191621930077: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:29:16,315 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:29:16,315 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:29:16,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:29:16,316 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-3048092522799696791: Routed from 90520738888109 2025-06-29 03:29:16,364 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 1-3048092522799696791: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:29:16,374 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:29:16,374 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:29:16,374 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:29:16,375 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-3048092522799696792: Routed from 90520738888109 2025-06-29 03:29:17,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f93162f3]) (logid:fd71b60f) No inactive management server node found 2025-06-29 03:29:17,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f93162f3]) (logid:fd71b60f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:17,467 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 1-3048092522799696792: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:29:17,469 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:29:17,469 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:29:17,469 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:29:17,470 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-1612007191621930078: Routed from 90520738888109 2025-06-29 03:29:18,376 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bbec327c]) (logid:104a1301) HA health check task is running... 2025-06-29 03:29:18,531 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1f7de7f2]) (logid:54492b4d) Seq 2-1612007191621930078: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:29:18,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e6a75fc]) (logid:87d19b3a) No inactive management server node found 2025-06-29 03:29:18,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e6a75fc]) (logid:87d19b3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:20,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8189: Seq 7-8189: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:29:20,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8189 2025-06-29 03:29:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33885515]) (logid:f2628d46) No inactive management server node found 2025-06-29 03:29:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33885515]) (logid:f2628d46) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:21,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77fbf03]) (logid:1628a56d) No inactive management server node found 2025-06-29 03:29:21,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77fbf03]) (logid:1628a56d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:22,378 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5f8af878]) (logid:175e4f25) HA health check task is running... 2025-06-29 03:29:23,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86fc22f6]) (logid:0bb84793) No inactive management server node found 2025-06-29 03:29:23,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86fc22f6]) (logid:0bb84793) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:24,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46b4f1a2]) (logid:d018206b) No inactive management server node found 2025-06-29 03:29:24,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46b4f1a2]) (logid:d018206b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:26,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7afcdd3e]) (logid:dd1eb7a3) No inactive management server node found 2025-06-29 03:29:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7afcdd3e]) (logid:dd1eb7a3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:26,379 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6179c317]) (logid:be8b7155) HA health check task is running... 2025-06-29 03:29:26,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4fdd55f4]) (logid:9219c536) Enabled non-edge zones available for scan: 1 2025-06-29 03:29:26,766 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4fdd55f4]) (logid:9219c536) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:29:26,767 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4fdd55f4]) (logid:9219c536) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:29:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-eaa25c09]) (logid:356cd606) Enabled non-edge zones available for scan: 1 2025-06-29 03:29:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-eaa25c09]) (logid:356cd606) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:29:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1e58f20]) (logid:fb041b02) No inactive management server node found 2025-06-29 03:29:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1e58f20]) (logid:fb041b02) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19b7dd1e]) (logid:0e64fab0) No inactive management server node found 2025-06-29 03:29:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19b7dd1e]) (logid:0e64fab0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:29:30,182 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8190: Seq 7-8190: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:29:30,203 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8190 2025-06-29 03:29:30,380 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c5bd94a1]) (logid:3a358854) HA health check task is running... 2025-06-29 03:29:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7be8fe3]) (logid:0c0830d7) No inactive management server node found 2025-06-29 03:29:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7be8fe3]) (logid:0c0830d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:31,024 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:29:31,025 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:29:31,026 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) Seq 2-3048092522799696793: Routed from 90520738888109 2025-06-29 03:29:31,028 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) Seq 1-3048092522799696793: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:29:31,030 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:29:31,031 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) Seq 2-1612007191621930079: Routed from 90520738888109 2025-06-29 03:29:31,034 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b573bd21]) (logid:6ce62951) Seq 2-1612007191621930079: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:29:32,331 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a18964b8]) (logid:d3b3ddac) No inactive management server node found 2025-06-29 03:29:32,331 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a18964b8]) (logid:d3b3ddac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5760c63e]) (logid:3a10fe2b) No inactive management server node found 2025-06-29 03:29:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5760c63e]) (logid:3a10fe2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:29:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:29:34,382 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8b76a0c4]) (logid:1534d882) HA health check task is running... 2025-06-29 03:29:35,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7d6452f]) (logid:995d9fec) No inactive management server node found 2025-06-29 03:29:35,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7d6452f]) (logid:995d9fec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-066783ae]) (logid:a522386c) No inactive management server node found 2025-06-29 03:29:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-066783ae]) (logid:a522386c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:38,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fd7eaf]) (logid:ae6c3b29) No inactive management server node found 2025-06-29 03:29:38,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fd7eaf]) (logid:ae6c3b29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:38,383 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7316f924]) (logid:84b9bfbe) HA health check task is running... 2025-06-29 03:29:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f998fe]) (logid:7556ef58) No inactive management server node found 2025-06-29 03:29:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f998fe]) (logid:7556ef58) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:40,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8191: Seq 7-8191: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:29:40,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8191 2025-06-29 03:29:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2e82a8ec]) (logid:ec585884) Found 0 routers to update status. 2025-06-29 03:29:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2e82a8ec]) (logid:ec585884) Found 0 VPC's to update Redundant State. 2025-06-29 03:29:41,004 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2e82a8ec]) (logid:ec585884) Found 0 networks to update RvR status. 2025-06-29 03:29:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-a8da4e07]) (logid:51c24c47) DbCollector is running... 2025-06-29 03:29:41,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79e8fcb5]) (logid:b47aabf6) No inactive management server node found 2025-06-29 03:29:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79e8fcb5]) (logid:b47aabf6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:29:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) ManagementServerCollector is running... 2025-06-29 03:29:42,154 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) DB properties were already loaded 2025-06-29 03:29:42,154 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:29:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Execution of process [4063889] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:29:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Exit value of process [4063889] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:29:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Process [4063889] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:29:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) usage install: null 2025-06-29 03:29:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Sessions active: 0 2025-06-29 03:29:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:29:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063892] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:29:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:29:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063897] for command [/bin/bash -c uname -r ]. 2025-06-29 03:29:42,183 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:29:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063898] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:29:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) system memory from /proc: 4887515136 2025-06-29 03:29:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:29:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063903] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:29:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) free memory from /proc: 486465536 2025-06-29 03:29:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:29:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063908] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:29:42,207 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) used memory from /proc: 769100 2025-06-29 03:29:42,207 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:29:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063909] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:29:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:29:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063910] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:29:42,224 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:29:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063911] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:29:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:29:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063918] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:29:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:29:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Successfully executed process [4063919] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail buffersdirect.count, 120 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail buffersdirect.used, 658750 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail buffersmapped.count, 7 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail buffersmapped.used, 798602 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail gcPS-Scavenge.count, 522 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail gcPS-Scavenge.time, 3781 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail jvmuptime, 44522325 2025-06-29 03:29:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memoryheap.committed, 358612992 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memoryheap.init, 77594624 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memoryheap.usage, 0.10939253021550355 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorynon-heap.max, -1 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorynon-heap.usage, -2.20751752E8 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorynon-heap.used, 220751752 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765383397760597 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30359166194554604 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.Metaspace.usage, 0.9859345237273631 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.PS-Eden-Space.usage, 0.029760010506567715 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.PS-Old-Gen.usage, 0.126824199127566 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorypools.PS-Survivor-Space.usage, 0.5338586171468099 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorytotal.committed, 583729152 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorytotal.init, 85262336 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorytotal.max, 1908932607 2025-06-29 03:29:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail memorytotal.used, 429624520 2025-06-29 03:29:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail threadsdeadlocks, [] 2025-06-29 03:29:42,250 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail threadsnew.count, 0 2025-06-29 03:29:42,251 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:29:42,252 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-473f40e5]) (logid:5c26aeb7) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:29:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":150509328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428855416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":208822968,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486465536,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.05,0.07],"systemCyclesUsage":[66377140,53816610,4588497391],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:29:42,253 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1483, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:29:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":150509328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428855416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":208822968,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486465536,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.05,0.07],"systemCyclesUsage":[66377140,53816610,4588497391],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:29:42,253 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:29:42,258 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-333:[ctx-523e1782]) (logid:e4b5078d) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:29:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":150509328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428855416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":208822968,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486465536,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.05,0.07],"systemCyclesUsage":[66377140,53816610,4588497391],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:29:42,258 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 03:29:42,258 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-016fef9d]) (logid:08c33476) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1483, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:29:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":150509328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44522246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":428855416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":208822968,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486465536,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.05,0.07],"systemCyclesUsage":[66377140,53816610,4588497391],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:29:42,384 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3648cb75]) (logid:5289acca) HA health check task is running... 2025-06-29 03:29:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64549ff2]) (logid:a8a0208c) No inactive management server node found 2025-06-29 03:29:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64549ff2]) (logid:a8a0208c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:44,327 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3916da0]) (logid:595abebf) No inactive management server node found 2025-06-29 03:29:44,327 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3916da0]) (logid:595abebf) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:29:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:29:45,496 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:29:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a757eada]) (logid:0ea5224e) No inactive management server node found 2025-06-29 03:29:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a757eada]) (logid:0ea5224e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-52f74c04]) (logid:30f3cd64) VM scheduler.poll is being called at 2025-06-29T01:30:00+0000 2025-06-29 03:29:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-52f74c04]) (logid:30f3cd64) Got 0 scheduled jobs to be executed at 2025-06-29T01:30:00+0000 2025-06-29 03:29:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-52f74c04]) (logid:30f3cd64) Cleaned up 0 VM scheduled job entries 2025-06-29 03:29:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-dbfd7bde]) (logid:dbd9f95f) VM Operation Thread Running 2025-06-29 03:29:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8827789b]) (logid:e7595729) ClusterDRS.poll is being called at 2025-06-29T01:30:00+0000 2025-06-29 03:29:46,343 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8827789b]) (logid:e7595729) Removed 0 old drs migration plans 2025-06-29 03:29:46,386 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-84406eec]) (logid:36cf6599) HA health check task is running... 2025-06-29 03:29:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933b48b6]) (logid:b73254e7) Resetting hosts suitable for reconnect 2025-06-29 03:29:47,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933b48b6]) (logid:b73254e7) Completed resetting hosts suitable for reconnect 2025-06-29 03:29:47,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933b48b6]) (logid:b73254e7) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:29:47,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933b48b6]) (logid:b73254e7) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:29:47,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933b48b6]) (logid:b73254e7) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:29:47,060 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-933b48b6]) (logid:b73254e7) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:29:47,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:47,240 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:47,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:29:47,245 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:29:47,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa4ac3ae]) (logid:2d3a7634) No inactive management server node found 2025-06-29 03:29:47,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa4ac3ae]) (logid:2d3a7634) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04ed5ba9]) (logid:11e6ceb1) No inactive management server node found 2025-06-29 03:29:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04ed5ba9]) (logid:11e6ceb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:50,183 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8193: Seq 7-8193: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:29:50,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8193 2025-06-29 03:29:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78d15b15]) (logid:075e2423) No inactive management server node found 2025-06-29 03:29:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78d15b15]) (logid:075e2423) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:50,387 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ecbe1f87]) (logid:eb46df67) HA health check task is running... 2025-06-29 03:29:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd1f8667]) (logid:cd1e7265) No inactive management server node found 2025-06-29 03:29:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd1f8667]) (logid:cd1e7265) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:53,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26e9fda1]) (logid:38c4af16) No inactive management server node found 2025-06-29 03:29:53,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26e9fda1]) (logid:38c4af16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:54,388 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9fe441ce]) (logid:d06de3a0) HA health check task is running... 2025-06-29 03:29:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48b7386e]) (logid:ed509f02) No inactive management server node found 2025-06-29 03:29:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48b7386e]) (logid:ed509f02) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:56,328 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6a94a42]) (logid:5c8f5ef0) No inactive management server node found 2025-06-29 03:29:56,328 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6a94a42]) (logid:5c8f5ef0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b4524981]) (logid:802e7146) Enabled non-edge zones available for scan: 1 2025-06-29 03:29:56,761 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b4524981]) (logid:802e7146) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:29:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b4524981]) (logid:802e7146) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:29:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d6c1a3a7]) (logid:fafc0e9d) Enabled non-edge zones available for scan: 1 2025-06-29 03:29:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d6c1a3a7]) (logid:fafc0e9d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:29:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac8cf8a5]) (logid:381a14e3) No inactive management server node found 2025-06-29 03:29:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac8cf8a5]) (logid:381a14e3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:29:57,964 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-983df6d9]) (logid:f0fd61a1) AutoScaling Monitor is running... 2025-06-29 03:29:58,091 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b08fa702]) (logid:ce2353e3) Starting to remove all volume_stats rows older than [Sat Jun 28 15:29:58 CEST 2025]. 2025-06-29 03:29:58,092 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b08fa702]) (logid:ce2353e3) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:29:58 CEST 2025]. 2025-06-29 03:29:58,389 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b6a63ec7]) (logid:1fd8c291) HA health check task is running... 2025-06-29 03:29:59,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b550ab4]) (logid:f4c5ac04) No inactive management server node found 2025-06-29 03:29:59,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b550ab4]) (logid:f4c5ac04) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:00,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8194: Seq 7-8194: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:00,193 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8194 2025-06-29 03:30:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0cbede4]) (logid:0777a406) No inactive management server node found 2025-06-29 03:30:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0cbede4]) (logid:0777a406) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adfa6194]) (logid:39b5503f) No inactive management server node found 2025-06-29 03:30:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adfa6194]) (logid:39b5503f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:02,390 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9d724331]) (logid:3b5bc076) HA health check task is running... 2025-06-29 03:30:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de261b67]) (logid:2eafe661) No inactive management server node found 2025-06-29 03:30:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de261b67]) (logid:2eafe661) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:05,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b56b8440]) (logid:caf86666) No inactive management server node found 2025-06-29 03:30:05,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b56b8440]) (logid:caf86666) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:06,391 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-52d58f18]) (logid:2cddb368) HA health check task is running... 2025-06-29 03:30:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c097077]) (logid:e3082e4d) No inactive management server node found 2025-06-29 03:30:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c097077]) (logid:e3082e4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:06,924 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:30:06,928 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:30:06,930 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) Seq 2-3048092522799696794: Routed from 90520738888109 2025-06-29 03:30:06,984 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) Seq 1-3048092522799696794: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:30:07,020 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:30:07,021 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) Seq 2-1612007191621930080: Routed from 90520738888109 2025-06-29 03:30:07,067 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-7b1117aa]) (logid:79f1a9d6) Seq 2-1612007191621930080: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:30:08,329 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c58621fe]) (logid:c02067fe) No inactive management server node found 2025-06-29 03:30:08,329 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c58621fe]) (logid:c02067fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9ab3b13]) (logid:897c1016) No inactive management server node found 2025-06-29 03:30:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9ab3b13]) (logid:897c1016) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:10,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8195: Seq 7-8195: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:10,195 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:30:10,206 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8195 2025-06-29 03:30:10,393 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c2200e90]) (logid:2a189b4c) HA health check task is running... 2025-06-29 03:30:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2c26112]) (logid:13722723) Found 0 routers to update status. 2025-06-29 03:30:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2c26112]) (logid:13722723) Found 0 VPC's to update Redundant State. 2025-06-29 03:30:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2c26112]) (logid:13722723) Found 0 networks to update RvR status. 2025-06-29 03:30:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daa92cff]) (logid:8aaa7856) No inactive management server node found 2025-06-29 03:30:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daa92cff]) (logid:8aaa7856) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:12,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-108695e1]) (logid:1656dd82) No inactive management server node found 2025-06-29 03:30:12,831 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-108695e1]) (logid:1656dd82) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:14,272 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-9dc102f1]) (logid:bf4781a7) Starting to remove all vm_stats rows older than [Sat Jun 28 15:30:14 CEST 2025]. 2025-06-29 03:30:14,310 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-9dc102f1]) (logid:bf4781a7) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:30:14 CEST 2025]. 2025-06-29 03:30:14,325 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c2e16dd]) (logid:560e8b96) No inactive management server node found 2025-06-29 03:30:14,325 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c2e16dd]) (logid:560e8b96) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:30:14,394 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e939c0c5]) (logid:c4a23b50) HA health check task is running... 2025-06-29 03:30:15,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8196: Seq 7-8196: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:15,206 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8196 2025-06-29 03:30:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d322e6d]) (logid:c208bae2) No inactive management server node found 2025-06-29 03:30:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d322e6d]) (logid:c208bae2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc59c94c]) (logid:03c747e0) No inactive management server node found 2025-06-29 03:30:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc59c94c]) (logid:03c747e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:18,395 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e0c2e03f]) (logid:c95acdab) HA health check task is running... 2025-06-29 03:30:18,532 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) StorageCollector is running... 2025-06-29 03:30:18,538 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:30:18,538 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:30:18,538 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:30:18,539 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 2-6717118844223095505: Routed from 90520738888109 2025-06-29 03:30:18,590 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 4-6717118844223095505: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:30:18,594 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:30:18,594 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:30:18,594 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:30:18,595 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 2-3048092522799696795: Routed from 90520738888109 2025-06-29 03:30:18,642 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 1-3048092522799696795: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:30:18,653 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:30:18,653 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:30:18,653 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:30:18,654 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 2-3048092522799696796: Routed from 90520738888109 2025-06-29 03:30:18,741 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 1-3048092522799696796: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:30:18,751 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:30:18,752 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:30:18,752 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:30:18,753 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 2-3048092522799696797: Routed from 90520738888109 2025-06-29 03:30:18,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-699e1095]) (logid:d2e294f1) No inactive management server node found 2025-06-29 03:30:18,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-699e1095]) (logid:d2e294f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:19,844 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 1-3048092522799696797: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:30:19,846 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:30:19,846 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:30:19,846 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:30:19,847 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 2-1612007191621930081: Routed from 90520738888109 2025-06-29 03:30:20,329 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ef2a13b]) (logid:3cfabb7b) No inactive management server node found 2025-06-29 03:30:20,329 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ef2a13b]) (logid:3cfabb7b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:20,865 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e122d6f4]) (logid:4d21283b) Seq 2-1612007191621930081: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:30:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11ee9231]) (logid:3323877f) No inactive management server node found 2025-06-29 03:30:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11ee9231]) (logid:3323877f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:22,397 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-77db6901]) (logid:9d0c9a08) HA health check task is running... 2025-06-29 03:30:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8045c84c]) (logid:ee7f09d9) No inactive management server node found 2025-06-29 03:30:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8045c84c]) (logid:ee7f09d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-587bc69d]) (logid:8c5e3f5f) No inactive management server node found 2025-06-29 03:30:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-587bc69d]) (logid:8c5e3f5f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:25,185 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8197: Seq 7-8197: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:25,205 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8197 2025-06-29 03:30:26,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43be0e1f]) (logid:90355bba) No inactive management server node found 2025-06-29 03:30:26,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43be0e1f]) (logid:90355bba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:26,398 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-072582f9]) (logid:04df2915) HA health check task is running... 2025-06-29 03:30:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-60e66108]) (logid:6361ac82) Enabled non-edge zones available for scan: 1 2025-06-29 03:30:26,763 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-60e66108]) (logid:6361ac82) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:30:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-60e66108]) (logid:6361ac82) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:30:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-abbeee69]) (logid:76035c98) Enabled non-edge zones available for scan: 1 2025-06-29 03:30:26,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-abbeee69]) (logid:76035c98) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:30:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c3d564]) (logid:433248cd) No inactive management server node found 2025-06-29 03:30:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c3d564]) (logid:433248cd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42b06889]) (logid:804b9c5b) No inactive management server node found 2025-06-29 03:30:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42b06889]) (logid:804b9c5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:30,400 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-945799dd]) (logid:e30b96c4) HA health check task is running... 2025-06-29 03:30:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cec1c375]) (logid:4190879d) No inactive management server node found 2025-06-29 03:30:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cec1c375]) (logid:4190879d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:31,036 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:30:31,037 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:30:31,038 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) Seq 2-3048092522799696798: Routed from 90520738888109 2025-06-29 03:30:31,040 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) Seq 1-3048092522799696798: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:30:31,041 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:30:31,042 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) Seq 2-1612007191621930082: Routed from 90520738888109 2025-06-29 03:30:31,045 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-2c7e2fbd]) (logid:de360552) Seq 2-1612007191621930082: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:30:32,339 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c10c39f]) (logid:e6c8cacd) No inactive management server node found 2025-06-29 03:30:32,339 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c10c39f]) (logid:e6c8cacd) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:33,835 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-633ec369]) (logid:c75b8ba6) No inactive management server node found 2025-06-29 03:30:33,835 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-633ec369]) (logid:c75b8ba6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:30:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:30:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:30:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:30:34,401 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-844cbf8d]) (logid:0261d9ab) HA health check task is running... 2025-06-29 03:30:35,185 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8198: Seq 7-8198: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:35,206 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8198 2025-06-29 03:30:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c10345d9]) (logid:a21cb1bc) No inactive management server node found 2025-06-29 03:30:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c10345d9]) (logid:a21cb1bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b666798c]) (logid:cfc4b725) No inactive management server node found 2025-06-29 03:30:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b666798c]) (logid:cfc4b725) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:38,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96b0a07a]) (logid:97dd86ae) No inactive management server node found 2025-06-29 03:30:38,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96b0a07a]) (logid:97dd86ae) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:30:38,402 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f22af435]) (logid:2b5f4d7d) HA health check task is running... 2025-06-29 03:30:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62a69737]) (logid:2cbb2648) No inactive management server node found 2025-06-29 03:30:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62a69737]) (logid:2cbb2648) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-495db84c]) (logid:a65fc61f) Found 0 routers to update status. 2025-06-29 03:30:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-495db84c]) (logid:a65fc61f) Found 0 VPC's to update Redundant State. 2025-06-29 03:30:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-495db84c]) (logid:a65fc61f) Found 0 networks to update RvR status. 2025-06-29 03:30:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-52d27434]) (logid:c6c1db72) DbCollector is running... 2025-06-29 03:30:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc2401d8]) (logid:3a84d710) No inactive management server node found 2025-06-29 03:30:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc2401d8]) (logid:3a84d710) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) ManagementServerCollector is running... 2025-06-29 03:30:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) DB properties were already loaded 2025-06-29 03:30:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:30:42,167 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Execution of process [4063941] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:30:42,167 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Exit value of process [4063941] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:30:42,167 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Process [4063941] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:30:42,167 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) usage install: null 2025-06-29 03:30:42,167 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Sessions active: 0 2025-06-29 03:30:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:30:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063944] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:30:42,175 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:30:42,180 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063949] for command [/bin/bash -c uname -r ]. 2025-06-29 03:30:42,181 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:30:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063950] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:30:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) system memory from /proc: 4887515136 2025-06-29 03:30:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:30:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063955] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:30:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) free memory from /proc: 494792704 2025-06-29 03:30:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:30:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063960] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:30:42,206 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) used memory from /proc: 769100 2025-06-29 03:30:42,206 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:30:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063961] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:30:42,213 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:30:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063962] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:30:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:30:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063963] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:30:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:30:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063970] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:30:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:30:42,243 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Successfully executed process [4063971] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail buffersdirect.count, 120 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail buffersdirect.used, 658750 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail buffersmapped.count, 7 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail buffersmapped.used, 798602 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail gcPS-Scavenge.count, 523 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail gcPS-Scavenge.time, 3788 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail jvmuptime, 44582323 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memoryheap.committed, 358088704 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memoryheap.init, 77594624 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memoryheap.usage, 0.10675284771498858 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorynon-heap.max, -1 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorynon-heap.usage, -2.20751752E8 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorynon-heap.used, 220751752 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765383397760597 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30359166194554604 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.Metaspace.usage, 0.9859345237273631 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.PS-Eden-Space.usage, 0.018146995347620685 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.PS-Old-Gen.usage, 0.12686424856000836 2025-06-29 03:30:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorypools.PS-Survivor-Space.usage, 0.7989176874575408 2025-06-29 03:30:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorytotal.committed, 583204864 2025-06-29 03:30:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorytotal.init, 85262336 2025-06-29 03:30:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorytotal.max, 1908932607 2025-06-29 03:30:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail memorytotal.used, 424596192 2025-06-29 03:30:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail threadsdeadlocks, [] 2025-06-29 03:30:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail threadsnew.count, 0 2025-06-29 03:30:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:30:42,251 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-3c478047]) (logid:69f0b504) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:30:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":155071256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44582245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423776752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203783992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494792704,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.06],"systemCyclesUsage":[66377477,53816943,4588520470],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:30:42,252 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1485, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:30:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":155071256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44582245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423776752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203783992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494792704,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.06],"systemCyclesUsage":[66377477,53816943,4588520470],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:30:42,252 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:30:42,257 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-333:[ctx-ca3f6226]) (logid:461f2ff0) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:30:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":155071256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44582245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423776752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203783992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494792704,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.06],"systemCyclesUsage":[66377477,53816943,4588520470],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:30:42,257 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:30:42,257 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 1485, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:30:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358088704,"freeJvmMemoryBytes":155071256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44582245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":423776752,"totalCommitted":583204864,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203783992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494792704,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.06],"systemCyclesUsage":[66377477,53816943,4588520470],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:30:42,403 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-08afb07a]) (logid:5ec0a9a0) HA health check task is running... 2025-06-29 03:30:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d55e6cd0]) (logid:cd22072e) No inactive management server node found 2025-06-29 03:30:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d55e6cd0]) (logid:cd22072e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:44,330 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-566de265]) (logid:653c4c36) No inactive management server node found 2025-06-29 03:30:44,330 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-566de265]) (logid:653c4c36) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:45,187 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8199: Seq 7-8199: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:45,207 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8199 2025-06-29 03:30:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:30:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:30:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff7666b2]) (logid:ad9d65fe) No inactive management server node found 2025-06-29 03:30:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff7666b2]) (logid:ad9d65fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-71f1cb10]) (logid:cf285d56) VM scheduler.poll is being called at 2025-06-29T01:31:00+0000 2025-06-29 03:30:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-71f1cb10]) (logid:cf285d56) Got 0 scheduled jobs to be executed at 2025-06-29T01:31:00+0000 2025-06-29 03:30:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-71f1cb10]) (logid:cf285d56) Cleaned up 0 VM scheduled job entries 2025-06-29 03:30:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d1d5dacc]) (logid:a8cb4a98) VM Operation Thread Running 2025-06-29 03:30:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-87ad90cf]) (logid:cc716b43) ClusterDRS.poll is being called at 2025-06-29T01:31:00+0000 2025-06-29 03:30:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-87ad90cf]) (logid:cc716b43) Removed 0 old drs migration plans 2025-06-29 03:30:46,404 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a39b81f7]) (logid:7f34f450) HA health check task is running... 2025-06-29 03:30:47,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:30:47,224 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:30:47,226 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:30:47,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:30:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52d80a86]) (logid:efabda96) No inactive management server node found 2025-06-29 03:30:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52d80a86]) (logid:efabda96) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:48,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1733c49e]) (logid:89ad77f1) No inactive management server node found 2025-06-29 03:30:48,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1733c49e]) (logid:89ad77f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7d43978]) (logid:ba5c338a) No inactive management server node found 2025-06-29 03:30:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7d43978]) (logid:ba5c338a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:50,406 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7a84f8cd]) (logid:eec7bdd8) HA health check task is running... 2025-06-29 03:30:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-488f6404]) (logid:5e07b9e1) No inactive management server node found 2025-06-29 03:30:51,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-488f6404]) (logid:5e07b9e1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:53,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6d4f52a]) (logid:f7d5a784) No inactive management server node found 2025-06-29 03:30:53,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6d4f52a]) (logid:f7d5a784) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:54,407 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-87aacefd]) (logid:077f9683) HA health check task is running... 2025-06-29 03:30:54,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d6b404d]) (logid:c00a0b46) No inactive management server node found 2025-06-29 03:30:54,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d6b404d]) (logid:c00a0b46) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:55,187 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8201: Seq 7-8201: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:30:55,208 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8201 2025-06-29 03:30:56,340 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a6a0ac]) (logid:97473d0e) No inactive management server node found 2025-06-29 03:30:56,340 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a6a0ac]) (logid:97473d0e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:30:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-32fcb880]) (logid:e977e5cf) Enabled non-edge zones available for scan: 1 2025-06-29 03:30:56,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-32fcb880]) (logid:e977e5cf) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:30:56,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-32fcb880]) (logid:e977e5cf) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:30:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9e30aeee]) (logid:ee5a61e7) Enabled non-edge zones available for scan: 1 2025-06-29 03:30:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9e30aeee]) (logid:ee5a61e7) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:30:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc8409a3]) (logid:0fdfe229) No inactive management server node found 2025-06-29 03:30:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc8409a3]) (logid:0fdfe229) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:30:57,966 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-caf1bd81]) (logid:1d273792) AutoScaling Monitor is running... 2025-06-29 03:30:58,094 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-458753f7]) (logid:cd44bcf0) Starting to remove all volume_stats rows older than [Sat Jun 28 15:30:58 CEST 2025]. 2025-06-29 03:30:58,094 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-458753f7]) (logid:cd44bcf0) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:30:58 CEST 2025]. 2025-06-29 03:30:58,408 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-91d44a26]) (logid:48840cfe) HA health check task is running... 2025-06-29 03:30:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c615e0ab]) (logid:81766d46) No inactive management server node found 2025-06-29 03:30:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c615e0ab]) (logid:81766d46) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:00,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b2f63a]) (logid:508c217f) No inactive management server node found 2025-06-29 03:31:00,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b2f63a]) (logid:508c217f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:02,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0faf6b29]) (logid:a71ed3c1) No inactive management server node found 2025-06-29 03:31:02,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0faf6b29]) (logid:a71ed3c1) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:31:02,409 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bc45cabc]) (logid:230b6806) HA health check task is running... 2025-06-29 03:31:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b6e2b8d]) (logid:393070ef) No inactive management server node found 2025-06-29 03:31:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b6e2b8d]) (logid:393070ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:05,188 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8202: Seq 7-8202: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:31:05,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8202 2025-06-29 03:31:05,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f306da7]) (logid:6d32a42b) No inactive management server node found 2025-06-29 03:31:05,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f306da7]) (logid:6d32a42b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:31:06,411 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-87a63457]) (logid:53664030) HA health check task is running... 2025-06-29 03:31:06,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9298963c]) (logid:82dc802a) No inactive management server node found 2025-06-29 03:31:06,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9298963c]) (logid:82dc802a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:07,123 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:31:07,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:31:07,127 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) Seq 2-3048092522799696799: Routed from 90520738888109 2025-06-29 03:31:07,180 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) Seq 1-3048092522799696799: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:31:07,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:31:07,229 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) Seq 2-1612007191621930083: Routed from 90520738888109 2025-06-29 03:31:07,286 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3b04aa9b]) (logid:8f3b79b2) Seq 2-1612007191621930083: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:31:08,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0ab8fde]) (logid:b3e13329) No inactive management server node found 2025-06-29 03:31:08,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0ab8fde]) (logid:b3e13329) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24e92dd2]) (logid:e06b6ff9) No inactive management server node found 2025-06-29 03:31:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24e92dd2]) (logid:e06b6ff9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:10,412 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f0be5bed]) (logid:5816f52c) HA health check task is running... 2025-06-29 03:31:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b11b98ef]) (logid:55114468) Found 0 routers to update status. 2025-06-29 03:31:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b11b98ef]) (logid:55114468) Found 0 VPC's to update Redundant State. 2025-06-29 03:31:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b11b98ef]) (logid:55114468) Found 0 networks to update RvR status. 2025-06-29 03:31:11,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e406dd96]) (logid:6c476365) No inactive management server node found 2025-06-29 03:31:11,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e406dd96]) (logid:6c476365) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:12,827 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcea30cf]) (logid:d05154c1) No inactive management server node found 2025-06-29 03:31:12,827 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcea30cf]) (logid:d05154c1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:14,312 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-58d78b53]) (logid:dd3c563b) Starting to remove all vm_stats rows older than [Sat Jun 28 15:31:14 CEST 2025]. 2025-06-29 03:31:14,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c14914af]) (logid:75e2612e) No inactive management server node found 2025-06-29 03:31:14,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c14914af]) (logid:75e2612e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:14,350 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-58d78b53]) (logid:dd3c563b) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:31:14 CEST 2025]. 2025-06-29 03:31:14,413 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ffe75b36]) (logid:39db9b0c) HA health check task is running... 2025-06-29 03:31:15,187 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8203: Seq 7-8203: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:31:15,208 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8203 2025-06-29 03:31:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cfc4d70]) (logid:85b29162) No inactive management server node found 2025-06-29 03:31:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cfc4d70]) (logid:85b29162) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:17,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0027328f]) (logid:9194a0a5) Resetting hosts suitable for reconnect 2025-06-29 03:31:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0027328f]) (logid:9194a0a5) Completed resetting hosts suitable for reconnect 2025-06-29 03:31:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0027328f]) (logid:9194a0a5) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:31:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0027328f]) (logid:9194a0a5) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:31:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0027328f]) (logid:9194a0a5) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:31:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0027328f]) (logid:9194a0a5) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:31:17,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b965138d]) (logid:1d7d26c4) No inactive management server node found 2025-06-29 03:31:17,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b965138d]) (logid:1d7d26c4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:18,414 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9e828d37]) (logid:280a44db) HA health check task is running... 2025-06-29 03:31:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-230d656c]) (logid:11c53cdd) No inactive management server node found 2025-06-29 03:31:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-230d656c]) (logid:11c53cdd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:20,342 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f85b3fa2]) (logid:0439ba48) No inactive management server node found 2025-06-29 03:31:20,342 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f85b3fa2]) (logid:0439ba48) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:31:20,865 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) StorageCollector is running... 2025-06-29 03:31:20,870 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:31:20,870 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:31:20,870 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:31:20,871 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-6717118844223095506: Routed from 90520738888109 2025-06-29 03:31:20,924 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 4-6717118844223095506: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:31:20,928 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:31:20,928 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:31:20,928 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:31:20,929 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-1612007191621930084: Routed from 90520738888109 2025-06-29 03:31:20,940 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-1612007191621930084: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:31:20,950 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:31:20,950 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:31:20,950 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:31:20,951 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-3048092522799696800: Routed from 90520738888109 2025-06-29 03:31:20,999 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 1-3048092522799696800: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:31:21,011 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:31:21,011 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:31:21,011 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:31:21,012 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-3048092522799696801: Routed from 90520738888109 2025-06-29 03:31:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf1e652c]) (logid:2244525a) No inactive management server node found 2025-06-29 03:31:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf1e652c]) (logid:2244525a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:22,105 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 1-3048092522799696801: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:31:22,107 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:31:22,107 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:31:22,107 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:31:22,107 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-1612007191621930085: Routed from 90520738888109 2025-06-29 03:31:22,416 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-282e53cd]) (logid:4a834d9f) HA health check task is running... 2025-06-29 03:31:23,171 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-8c90f9ff]) (logid:6bd7b620) Seq 2-1612007191621930085: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:31:23,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2db50bb]) (logid:eca9339f) No inactive management server node found 2025-06-29 03:31:23,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2db50bb]) (logid:eca9339f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fa71780]) (logid:23cba21c) No inactive management server node found 2025-06-29 03:31:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fa71780]) (logid:23cba21c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:25,188 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8204: Seq 7-8204: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:31:25,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8204 2025-06-29 03:31:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb137a3c]) (logid:b2051ee2) No inactive management server node found 2025-06-29 03:31:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb137a3c]) (logid:b2051ee2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:26,417 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-89fcafef]) (logid:22754508) HA health check task is running... 2025-06-29 03:31:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ad50b9fd]) (logid:08d197cc) Enabled non-edge zones available for scan: 1 2025-06-29 03:31:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ad50b9fd]) (logid:08d197cc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:31:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ad50b9fd]) (logid:08d197cc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:31:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a22ef9a8]) (logid:1b5d0f09) Enabled non-edge zones available for scan: 1 2025-06-29 03:31:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a22ef9a8]) (logid:1b5d0f09) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:31:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44c0e516]) (logid:fc5a60f4) No inactive management server node found 2025-06-29 03:31:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44c0e516]) (logid:fc5a60f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53871448]) (logid:a7a3459e) No inactive management server node found 2025-06-29 03:31:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53871448]) (logid:a7a3459e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:30,418 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a41f635e]) (logid:0644f9bd) HA health check task is running... 2025-06-29 03:31:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8293f0dc]) (logid:aa753110) No inactive management server node found 2025-06-29 03:31:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8293f0dc]) (logid:aa753110) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:31,047 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:31:31,048 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:31:31,048 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) Seq 2-3048092522799696802: Routed from 90520738888109 2025-06-29 03:31:31,050 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) Seq 1-3048092522799696802: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:31:31,051 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:31:31,052 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) Seq 2-1612007191621930086: Routed from 90520738888109 2025-06-29 03:31:31,055 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-dc8dd4cc]) (logid:c4700452) Seq 2-1612007191621930086: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:31:32,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f452811]) (logid:9b3ab35d) No inactive management server node found 2025-06-29 03:31:32,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f452811]) (logid:9b3ab35d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb78ee23]) (logid:312bc1a0) No inactive management server node found 2025-06-29 03:31:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb78ee23]) (logid:312bc1a0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:31:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:31:33,982 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:31:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:31:34,419 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-185ca7b1]) (logid:a86a44c9) HA health check task is running... 2025-06-29 03:31:35,189 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8205: Seq 7-8205: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:31:35,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8205 2025-06-29 03:31:35,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-611e0db2]) (logid:43c4e451) No inactive management server node found 2025-06-29 03:31:35,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-611e0db2]) (logid:43c4e451) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fac935e]) (logid:da87d084) No inactive management server node found 2025-06-29 03:31:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fac935e]) (logid:da87d084) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:38,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-079e9883]) (logid:35352da8) No inactive management server node found 2025-06-29 03:31:38,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-079e9883]) (logid:35352da8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:38,420 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ba3bd07d]) (logid:4d8e40d8) HA health check task is running... 2025-06-29 03:31:39,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99dd0da5]) (logid:f69760df) No inactive management server node found 2025-06-29 03:31:39,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99dd0da5]) (logid:f69760df) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ed5adf5e]) (logid:47330f56) Found 0 routers to update status. 2025-06-29 03:31:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ed5adf5e]) (logid:47330f56) Found 0 VPC's to update Redundant State. 2025-06-29 03:31:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ed5adf5e]) (logid:47330f56) Found 0 networks to update RvR status. 2025-06-29 03:31:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-83995c19]) (logid:a5820375) DbCollector is running... 2025-06-29 03:31:41,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c4db257]) (logid:334407ed) No inactive management server node found 2025-06-29 03:31:41,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c4db257]) (logid:334407ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) ManagementServerCollector is running... 2025-06-29 03:31:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) DB properties were already loaded 2025-06-29 03:31:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:31:42,170 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Execution of process [4063976] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:31:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Exit value of process [4063976] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:31:42,170 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Process [4063976] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:31:42,170 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) usage install: null 2025-06-29 03:31:42,170 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Sessions active: 0 2025-06-29 03:31:42,172 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:31:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063979] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:31:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:31:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063984] for command [/bin/bash -c uname -r ]. 2025-06-29 03:31:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:31:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063985] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:31:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) system memory from /proc: 4887515136 2025-06-29 03:31:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:31:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063990] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:31:42,194 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) free memory from /proc: 487763968 2025-06-29 03:31:42,194 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:31:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063995] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:31:42,211 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) used memory from /proc: 769100 2025-06-29 03:31:42,211 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:31:42,218 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063996] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:31:42,218 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:31:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063997] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:31:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:31:42,246 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4063998] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:31:42,246 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:31:42,250 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4064005] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:31:42,250 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:31:42,256 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Successfully executed process [4064006] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:31:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail buffersdirect.count, 120 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail buffersdirect.used, 658750 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail buffersmapped.count, 7 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail buffersmapped.used, 798602 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail gcPS-Scavenge.count, 524 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail gcPS-Scavenge.time, 3794 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail jvmuptime, 44642336 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memoryheap.committed, 358612992 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memoryheap.init, 77594624 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memoryheap.usage, 0.10097365574468724 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorynon-heap.max, -1 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorynon-heap.usage, -2.20751752E8 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorynon-heap.used, 220751752 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:31:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765383397760597 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30359166194554604 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.Metaspace.usage, 0.9859345237273631 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.PS-Eden-Space.usage, 0.003694550232536067 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.PS-Old-Gen.usage, 0.12688141260248365 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorypools.PS-Survivor-Space.usage, 0.6809972127278646 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorytotal.committed, 583729152 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorytotal.init, 85262336 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorytotal.max, 1908932607 2025-06-29 03:31:42,260 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail memorytotal.used, 413567632 2025-06-29 03:31:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail threadsdeadlocks, [] 2025-06-29 03:31:42,261 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail threadsnew.count, 0 2025-06-29 03:31:42,262 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:31:42,264 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-e5efa8e2]) (logid:61f7b5a5) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:31:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":166628984,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44642247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":1.29,"totalInit":85262336,"totalUsed":412741600,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192757720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487763968,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.29,0.36,0.17],"systemCyclesUsage":[66377792,53817241,4588543603],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:31:42,265 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1487, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:31:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":166628984,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44642247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":1.29,"totalInit":85262336,"totalUsed":412741600,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192757720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487763968,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.29,0.36,0.17],"systemCyclesUsage":[66377792,53817241,4588543603],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:31:42,265 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:31:42,268 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 3 ms 2025-06-29 03:31:42,268 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-728279bd]) (logid:828dd140) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 3ms. agent: 0, pdu seq: 1487, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:31:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":166628984,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44642247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":1.29,"totalInit":85262336,"totalUsed":412741600,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192757720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487763968,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.29,0.36,0.17],"systemCyclesUsage":[66377792,53817241,4588543603],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:31:42,268 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-334:[ctx-749c6483]) (logid:8ef187f5) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:31:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":166628984,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44642247,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":1.29,"totalInit":85262336,"totalUsed":412741600,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":192757720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":487763968,"systemMemoryUsed":769100,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.29,0.36,0.17],"systemCyclesUsage":[66377792,53817241,4588543603],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:31:42,421 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9e9eb8f7]) (logid:7020fb2b) HA health check task is running... 2025-06-29 03:31:42,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14aa7fee]) (logid:40ebc8e8) No inactive management server node found 2025-06-29 03:31:42,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14aa7fee]) (logid:40ebc8e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:44,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6df3caa]) (logid:fdf98341) No inactive management server node found 2025-06-29 03:31:44,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6df3caa]) (logid:fdf98341) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:45,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8206: Seq 7-8206: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:31:45,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8206 2025-06-29 03:31:45,341 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:31:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:31:45,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ceded6cb]) (logid:8aaed78b) No inactive management server node found 2025-06-29 03:31:45,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ceded6cb]) (logid:8aaed78b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a2a54d05]) (logid:b6ecbc52) VM scheduler.poll is being called at 2025-06-29T01:32:00+0000 2025-06-29 03:31:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a2a54d05]) (logid:b6ecbc52) Got 0 scheduled jobs to be executed at 2025-06-29T01:32:00+0000 2025-06-29 03:31:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a2a54d05]) (logid:b6ecbc52) Cleaned up 0 VM scheduled job entries 2025-06-29 03:31:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-68b7841e]) (logid:1a2adf9c) VM Operation Thread Running 2025-06-29 03:31:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-24e6d378]) (logid:1197c11b) ClusterDRS.poll is being called at 2025-06-29T01:32:00+0000 2025-06-29 03:31:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-24e6d378]) (logid:1197c11b) Removed 0 old drs migration plans 2025-06-29 03:31:46,422 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-28605031]) (logid:369d0044) HA health check task is running... 2025-06-29 03:31:47,241 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:31:47,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:31:47,244 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:31:47,247 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:31:47,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d431f47]) (logid:4ae632fb) No inactive management server node found 2025-06-29 03:31:47,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d431f47]) (logid:4ae632fb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bf7f07f]) (logid:5083589e) No inactive management server node found 2025-06-29 03:31:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bf7f07f]) (logid:5083589e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:50,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dad863b]) (logid:d290ab89) No inactive management server node found 2025-06-29 03:31:50,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dad863b]) (logid:d290ab89) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:50,424 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-21843e9e]) (logid:49b802ea) HA health check task is running... 2025-06-29 03:31:51,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1eeb3ff]) (logid:0e442d3a) No inactive management server node found 2025-06-29 03:31:51,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1eeb3ff]) (logid:0e442d3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:53,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61e15645]) (logid:a2af1380) No inactive management server node found 2025-06-29 03:31:53,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61e15645]) (logid:a2af1380) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:54,425 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7edc0a8f]) (logid:29023382) HA health check task is running... 2025-06-29 03:31:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cc0efb3]) (logid:375cf80b) No inactive management server node found 2025-06-29 03:31:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cc0efb3]) (logid:375cf80b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:55,189 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8208: Seq 7-8208: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:31:55,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8208 2025-06-29 03:31:56,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27e41b67]) (logid:ccbfdc8e) No inactive management server node found 2025-06-29 03:31:56,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27e41b67]) (logid:ccbfdc8e) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:56,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1f34f713]) (logid:db26086c) Enabled non-edge zones available for scan: 1 2025-06-29 03:31:56,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1f34f713]) (logid:db26086c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:31:56,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1f34f713]) (logid:db26086c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:31:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8e26c9f2]) (logid:ab963fc9) Enabled non-edge zones available for scan: 1 2025-06-29 03:31:56,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8e26c9f2]) (logid:ab963fc9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:31:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-748c01d2]) (logid:6bc3f039) No inactive management server node found 2025-06-29 03:31:57,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-748c01d2]) (logid:6bc3f039) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:31:57,968 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-ca87de1c]) (logid:d2585613) AutoScaling Monitor is running... 2025-06-29 03:31:58,097 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-599a4946]) (logid:ab114c61) Starting to remove all volume_stats rows older than [Sat Jun 28 15:31:58 CEST 2025]. 2025-06-29 03:31:58,097 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-599a4946]) (logid:ab114c61) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:31:58 CEST 2025]. 2025-06-29 03:31:58,426 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d898d0ef]) (logid:238e63f1) HA health check task is running... 2025-06-29 03:31:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88f71684]) (logid:2bce4106) No inactive management server node found 2025-06-29 03:31:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88f71684]) (logid:2bce4106) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:00,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8209: Seq 7-8209: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:00,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8209 2025-06-29 03:32:00,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-771175f7]) (logid:25a0c670) No inactive management server node found 2025-06-29 03:32:00,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-771175f7]) (logid:25a0c670) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:02,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ec5154d]) (logid:b2746c86) No inactive management server node found 2025-06-29 03:32:02,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ec5154d]) (logid:b2746c86) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:02,427 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-61749e95]) (logid:35ffda63) HA health check task is running... 2025-06-29 03:32:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b1c8e13]) (logid:d69b0ce6) No inactive management server node found 2025-06-29 03:32:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b1c8e13]) (logid:d69b0ce6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:05,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec4ec182]) (logid:f4544d0f) No inactive management server node found 2025-06-29 03:32:05,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec4ec182]) (logid:f4544d0f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:32:06,428 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8e0c1d12]) (logid:5b0e2d9e) HA health check task is running... 2025-06-29 03:32:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-407d2436]) (logid:b6f04f16) No inactive management server node found 2025-06-29 03:32:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-407d2436]) (logid:b6f04f16) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:07,344 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:32:07,348 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:32:07,349 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) Seq 2-3048092522799696803: Routed from 90520738888109 2025-06-29 03:32:07,404 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) Seq 1-3048092522799696803: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:32:07,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:32:07,450 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) Seq 2-1612007191621930087: Routed from 90520738888109 2025-06-29 03:32:07,506 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-b78f0233]) (logid:5bdbeb10) Seq 2-1612007191621930087: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:32:08,339 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1491ef1]) (logid:1422ac89) No inactive management server node found 2025-06-29 03:32:08,339 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1491ef1]) (logid:1422ac89) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a825ba0f]) (logid:407b39f9) No inactive management server node found 2025-06-29 03:32:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a825ba0f]) (logid:407b39f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:10,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8210: Seq 7-8210: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:10,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8210 2025-06-29 03:32:10,429 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6e0aa689]) (logid:6516d7ef) HA health check task is running... 2025-06-29 03:32:10,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9c972eb9]) (logid:3f60ae6a) Found 0 routers to update status. 2025-06-29 03:32:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9c972eb9]) (logid:3f60ae6a) Found 0 VPC's to update Redundant State. 2025-06-29 03:32:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9c972eb9]) (logid:3f60ae6a) Found 0 networks to update RvR status. 2025-06-29 03:32:11,273 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-1612007191621926913: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:11,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-687c489d]) (logid:fb92acea) No inactive management server node found 2025-06-29 03:32:11,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-687c489d]) (logid:fb92acea) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52ffe603]) (logid:d978cb0d) No inactive management server node found 2025-06-29 03:32:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52ffe603]) (logid:d978cb0d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:14,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0787527]) (logid:b7fee22b) No inactive management server node found 2025-06-29 03:32:14,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0787527]) (logid:b7fee22b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:14,352 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-7c3f0c7f]) (logid:b9f40b7a) Starting to remove all vm_stats rows older than [Sat Jun 28 15:32:14 CEST 2025]. 2025-06-29 03:32:14,378 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-7c3f0c7f]) (logid:b9f40b7a) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:32:14 CEST 2025]. 2025-06-29 03:32:14,431 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a8095e1e]) (logid:0a9eedd3) HA health check task is running... 2025-06-29 03:32:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fc522c4]) (logid:fe88e19f) No inactive management server node found 2025-06-29 03:32:15,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fc522c4]) (logid:fe88e19f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:17,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08d29df7]) (logid:6b40f063) No inactive management server node found 2025-06-29 03:32:17,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08d29df7]) (logid:6b40f063) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:18,432 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c38827e6]) (logid:3a32732c) HA health check task is running... 2025-06-29 03:32:18,831 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8522cb4]) (logid:beb36a03) No inactive management server node found 2025-06-29 03:32:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8522cb4]) (logid:beb36a03) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:20,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8211: Seq 7-8211: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:20,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8211 2025-06-29 03:32:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-587d0246]) (logid:bce5fd41) No inactive management server node found 2025-06-29 03:32:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-587d0246]) (logid:bce5fd41) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:21,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a7b49d]) (logid:748722f6) No inactive management server node found 2025-06-29 03:32:21,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65a7b49d]) (logid:748722f6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:22,433 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b157ebdc]) (logid:38b3270c) HA health check task is running... 2025-06-29 03:32:23,171 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) StorageCollector is running... 2025-06-29 03:32:23,177 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:32:23,177 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:32:23,177 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:32:23,178 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-6717118844223095507: Routed from 90520738888109 2025-06-29 03:32:23,237 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 4-6717118844223095507: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:32:23,242 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:32:23,242 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:32:23,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:32:23,243 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-1612007191621930088: Routed from 90520738888109 2025-06-29 03:32:23,255 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-1612007191621930088: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:32:23,279 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:32:23,279 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:32:23,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:32:23,280 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-3048092522799696804: Routed from 90520738888109 2025-06-29 03:32:23,328 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 1-3048092522799696804: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:32:23,355 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-483b24c8]) (logid:93a5956a) No inactive management server node found 2025-06-29 03:32:23,355 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-483b24c8]) (logid:93a5956a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:23,358 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:32:23,358 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:32:23,358 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:32:23,359 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-3048092522799696805: Routed from 90520738888109 2025-06-29 03:32:24,450 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 1-3048092522799696805: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:32:24,452 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:32:24,452 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:32:24,452 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:32:24,453 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-1612007191621930089: Routed from 90520738888109 2025-06-29 03:32:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eef0f950]) (logid:df1d4866) No inactive management server node found 2025-06-29 03:32:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eef0f950]) (logid:df1d4866) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:25,512 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-fb5ac7ae]) (logid:f44b2e5d) Seq 2-1612007191621930089: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:32:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d36a41b]) (logid:598c0556) No inactive management server node found 2025-06-29 03:32:26,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d36a41b]) (logid:598c0556) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:26,434 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5e6d2920]) (logid:0c638640) HA health check task is running... 2025-06-29 03:32:26,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-775f7ddf]) (logid:b926710c) Enabled non-edge zones available for scan: 1 2025-06-29 03:32:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-775f7ddf]) (logid:b926710c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:32:26,763 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-775f7ddf]) (logid:b926710c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:32:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c4995632]) (logid:df982ba7) Enabled non-edge zones available for scan: 1 2025-06-29 03:32:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c4995632]) (logid:df982ba7) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:32:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22ff72c3]) (logid:fbe81be5) No inactive management server node found 2025-06-29 03:32:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22ff72c3]) (logid:fbe81be5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:29,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a6b1699]) (logid:4939af6b) No inactive management server node found 2025-06-29 03:32:29,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a6b1699]) (logid:4939af6b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:30,191 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8212: Seq 7-8212: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:30,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8212 2025-06-29 03:32:30,435 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7e104498]) (logid:d0f5130a) HA health check task is running... 2025-06-29 03:32:30,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28de4a87]) (logid:4282302d) No inactive management server node found 2025-06-29 03:32:30,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28de4a87]) (logid:4282302d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:31,057 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:32:31,058 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:32:31,059 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) Seq 2-3048092522799696806: Routed from 90520738888109 2025-06-29 03:32:31,061 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) Seq 1-3048092522799696806: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:32:31,062 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:32:31,063 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) Seq 2-1612007191621930090: Routed from 90520738888109 2025-06-29 03:32:31,066 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-6b364822]) (logid:37c34a76) Seq 2-1612007191621930090: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:32:32,348 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fe58729]) (logid:58896c33) No inactive management server node found 2025-06-29 03:32:32,348 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fe58729]) (logid:58896c33) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 3ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:33,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c52fdacf]) (logid:0b850c02) No inactive management server node found 2025-06-29 03:32:33,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c52fdacf]) (logid:0b850c02) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:32:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:32:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:32:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:32:34,436 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e7fcbe2b]) (logid:e05d6a45) HA health check task is running... 2025-06-29 03:32:35,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c93bde34]) (logid:9636442c) No inactive management server node found 2025-06-29 03:32:35,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c93bde34]) (logid:9636442c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aa5e4ce]) (logid:e83ba586) No inactive management server node found 2025-06-29 03:32:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aa5e4ce]) (logid:e83ba586) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8eae9dbe]) (logid:fd403cc2) No inactive management server node found 2025-06-29 03:32:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8eae9dbe]) (logid:fd403cc2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:38,438 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8ac0f3c5]) (logid:dcfc083f) HA health check task is running... 2025-06-29 03:32:39,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c544bb0]) (logid:cad9572c) No inactive management server node found 2025-06-29 03:32:39,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c544bb0]) (logid:cad9572c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:40,194 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8213: Seq 7-8213: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:40,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8213 2025-06-29 03:32:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f72ace6b]) (logid:7b39c986) Found 0 routers to update status. 2025-06-29 03:32:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f72ace6b]) (logid:7b39c986) Found 0 VPC's to update Redundant State. 2025-06-29 03:32:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f72ace6b]) (logid:7b39c986) Found 0 networks to update RvR status. 2025-06-29 03:32:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-2c2cc6bc]) (logid:8fc98ff4) DbCollector is running... 2025-06-29 03:32:41,338 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc828bac]) (logid:58e4415a) No inactive management server node found 2025-06-29 03:32:41,338 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc828bac]) (logid:58e4415a) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 5ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:42,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) ManagementServerCollector is running... 2025-06-29 03:32:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) DB properties were already loaded 2025-06-29 03:32:42,154 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:32:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Execution of process [4064020] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:32:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Exit value of process [4064020] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:32:42,169 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Process [4064020] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:32:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) usage install: null 2025-06-29 03:32:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Sessions active: 0 2025-06-29 03:32:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:32:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064023] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:32:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:32:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064028] for command [/bin/bash -c uname -r ]. 2025-06-29 03:32:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:32:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064029] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:32:42,188 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) system memory from /proc: 4887515136 2025-06-29 03:32:42,188 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:32:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064034] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:32:42,193 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) free memory from /proc: 486150144 2025-06-29 03:32:42,193 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:32:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064039] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:32:42,205 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) used memory from /proc: 769996 2025-06-29 03:32:42,205 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:32:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064040] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:32:42,212 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:32:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064041] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:32:42,225 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:32:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064042] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:32:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:32:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064049] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:32:42,237 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:32:42,244 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Successfully executed process [4064050] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail buffersdirect.count, 120 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail buffersdirect.used, 658750 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail buffersmapped.count, 7 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail buffersmapped.used, 798602 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail gcPS-Scavenge.count, 524 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail gcPS-Scavenge.time, 3794 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail jvmuptime, 44702324 2025-06-29 03:32:42,246 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memoryheap.committed, 358612992 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memoryheap.init, 77594624 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memoryheap.usage, 0.11321308206182625 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorynon-heap.max, -1 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorynon-heap.usage, -2.20751752E8 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorynon-heap.used, 220751752 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765383397760597 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30359166194554604 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.Metaspace.usage, 0.9859345237273631 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.PS-Eden-Space.usage, 0.03756336162917616 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.PS-Old-Gen.usage, 0.12688141260248365 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorypools.PS-Survivor-Space.usage, 0.6809972127278646 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorytotal.committed, 583729152 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorytotal.init, 85262336 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorytotal.max, 1908932607 2025-06-29 03:32:42,247 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail memorytotal.used, 436930384 2025-06-29 03:32:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail threadsdeadlocks, [] 2025-06-29 03:32:42,248 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail threadsnew.count, 0 2025-06-29 03:32:42,249 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:32:42,251 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-5e425061]) (logid:9293c090) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:32:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":143222328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44702246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.55,"totalInit":85262336,"totalUsed":436142416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":216147392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486150144,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.55,0.32,0.17],"systemCyclesUsage":[66378099,53817567,4588566713],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:32:42,251 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1489, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:32:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":143222328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44702246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.55,"totalInit":85262336,"totalUsed":436142416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":216147392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486150144,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.55,0.32,0.17],"systemCyclesUsage":[66378099,53817567,4588566713],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:32:42,251 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:32:42,255 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-334:[ctx-f40e7706]) (logid:acaed80f) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:32:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":143222328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44702246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.55,"totalInit":85262336,"totalUsed":436142416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":216147392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486150144,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.55,0.32,0.17],"systemCyclesUsage":[66378099,53817567,4588566713],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:32:42,255 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:32:42,256 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1489, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:32:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":358612992,"freeJvmMemoryBytes":143222328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44702246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.55,"totalInit":85262336,"totalUsed":436142416,"totalCommitted":583729152,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":216147392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":486150144,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.55,0.32,0.17],"systemCyclesUsage":[66378099,53817567,4588566713],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:32:42,439 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e278b64a]) (logid:aac20dfa) HA health check task is running... 2025-06-29 03:32:42,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9455bf84]) (logid:a7c39229) No inactive management server node found 2025-06-29 03:32:42,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9455bf84]) (logid:a7c39229) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:44,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6579e04]) (logid:0c958355) No inactive management server node found 2025-06-29 03:32:44,332 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6579e04]) (logid:0c958355) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:32:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:32:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-987146ad]) (logid:cda1f66d) No inactive management server node found 2025-06-29 03:32:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-987146ad]) (logid:cda1f66d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d32829c0]) (logid:5d98907b) VM scheduler.poll is being called at 2025-06-29T01:33:00+0000 2025-06-29 03:32:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d32829c0]) (logid:5d98907b) Got 0 scheduled jobs to be executed at 2025-06-29T01:33:00+0000 2025-06-29 03:32:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d32829c0]) (logid:5d98907b) Cleaned up 0 VM scheduled job entries 2025-06-29 03:32:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-7b19ab7d]) (logid:2aa2da96) VM Operation Thread Running 2025-06-29 03:32:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8f5988a9]) (logid:fa9f32f7) ClusterDRS.poll is being called at 2025-06-29T01:33:00+0000 2025-06-29 03:32:46,341 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-8f5988a9]) (logid:fa9f32f7) Removed 0 old drs migration plans 2025-06-29 03:32:46,440 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4148cb2a]) (logid:535c58c1) HA health check task is running... 2025-06-29 03:32:47,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17571555]) (logid:39052a7c) Resetting hosts suitable for reconnect 2025-06-29 03:32:47,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17571555]) (logid:39052a7c) Completed resetting hosts suitable for reconnect 2025-06-29 03:32:47,057 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17571555]) (logid:39052a7c) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:32:47,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17571555]) (logid:39052a7c) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:32:47,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17571555]) (logid:39052a7c) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:32:47,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-17571555]) (logid:39052a7c) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:32:47,286 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:32:47,288 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:32:47,289 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:32:47,292 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:32:47,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fec49bc0]) (logid:963b9dc5) No inactive management server node found 2025-06-29 03:32:47,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fec49bc0]) (logid:963b9dc5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:48,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2ee71c]) (logid:9c8300d7) No inactive management server node found 2025-06-29 03:32:48,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2ee71c]) (logid:9c8300d7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:50,192 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8215: Seq 7-8215: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:32:50,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8215 2025-06-29 03:32:50,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f2d3647]) (logid:9e8c8c12) No inactive management server node found 2025-06-29 03:32:50,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f2d3647]) (logid:9e8c8c12) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:50,441 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1543717a]) (logid:49ff2a5e) HA health check task is running... 2025-06-29 03:32:51,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-429ea96f]) (logid:db36ee97) No inactive management server node found 2025-06-29 03:32:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-429ea96f]) (logid:db36ee97) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:53,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe629adc]) (logid:e9647a52) No inactive management server node found 2025-06-29 03:32:53,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe629adc]) (logid:e9647a52) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:54,443 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f5d4de23]) (logid:32462778) HA health check task is running... 2025-06-29 03:32:54,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79f27514]) (logid:62f897fa) No inactive management server node found 2025-06-29 03:32:54,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79f27514]) (logid:62f897fa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:56,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ffb9dfa]) (logid:11a43346) No inactive management server node found 2025-06-29 03:32:56,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ffb9dfa]) (logid:11a43346) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7bb2082c]) (logid:a55eb4ac) Enabled non-edge zones available for scan: 1 2025-06-29 03:32:56,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-7bb2082c]) (logid:a55eb4ac) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:32:56,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7bb2082c]) (logid:a55eb4ac) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:32:56,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5adad2b1]) (logid:e47a491b) Enabled non-edge zones available for scan: 1 2025-06-29 03:32:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5adad2b1]) (logid:e47a491b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:32:57,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e03487d5]) (logid:475e4e3d) No inactive management server node found 2025-06-29 03:32:57,840 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e03487d5]) (logid:475e4e3d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:32:57,970 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-11d28874]) (logid:898b1603) AutoScaling Monitor is running... 2025-06-29 03:32:58,099 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-34c7e982]) (logid:c9134821) Starting to remove all volume_stats rows older than [Sat Jun 28 15:32:58 CEST 2025]. 2025-06-29 03:32:58,100 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-34c7e982]) (logid:c9134821) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:32:58 CEST 2025]. 2025-06-29 03:32:58,444 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9121a453]) (logid:1f6e443f) HA health check task is running... 2025-06-29 03:32:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78b95ab7]) (logid:4897bb5b) No inactive management server node found 2025-06-29 03:32:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78b95ab7]) (logid:4897bb5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:00,192 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8216: Seq 7-8216: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:33:00,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8216 2025-06-29 03:33:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c67974d5]) (logid:8d9b6148) No inactive management server node found 2025-06-29 03:33:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c67974d5]) (logid:8d9b6148) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:02,342 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3203c32b]) (logid:5985008d) No inactive management server node found 2025-06-29 03:33:02,342 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3203c32b]) (logid:5985008d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:02,445 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4de765b2]) (logid:662be354) HA health check task is running... 2025-06-29 03:33:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5c6dac5]) (logid:3e190ecb) No inactive management server node found 2025-06-29 03:33:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5c6dac5]) (logid:3e190ecb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:05,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca4f6811]) (logid:2c779d04) No inactive management server node found 2025-06-29 03:33:05,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca4f6811]) (logid:2c779d04) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:06,447 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2cc22431]) (logid:b81f7d03) HA health check task is running... 2025-06-29 03:33:06,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72506b67]) (logid:121bce99) No inactive management server node found 2025-06-29 03:33:06,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72506b67]) (logid:121bce99) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:07,560 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:33:07,564 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:33:07,566 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) Seq 2-3048092522799696807: Routed from 90520738888109 2025-06-29 03:33:07,618 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) Seq 1-3048092522799696807: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:33:07,665 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:33:07,666 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) Seq 2-1612007191621930091: Routed from 90520738888109 2025-06-29 03:33:07,714 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-740656fd]) (logid:e69a2a4e) Seq 2-1612007191621930091: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:33:08,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e985995]) (logid:c50f2c0a) No inactive management server node found 2025-06-29 03:33:08,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e985995]) (logid:c50f2c0a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f7e960a]) (logid:44191969) No inactive management server node found 2025-06-29 03:33:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f7e960a]) (logid:44191969) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:10,193 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8217: Seq 7-8217: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:33:10,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8217 2025-06-29 03:33:10,448 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-be19acd9]) (logid:104b987e) HA health check task is running... 2025-06-29 03:33:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-63888fdc]) (logid:7c4b5255) Found 0 routers to update status. 2025-06-29 03:33:11,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-63888fdc]) (logid:7c4b5255) Found 0 VPC's to update Redundant State. 2025-06-29 03:33:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-63888fdc]) (logid:7c4b5255) Found 0 networks to update RvR status. 2025-06-29 03:33:11,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72ccbb97]) (logid:5d7731f8) No inactive management server node found 2025-06-29 03:33:11,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72ccbb97]) (logid:5d7731f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:12,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e14344eb]) (logid:61204b11) No inactive management server node found 2025-06-29 03:33:12,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e14344eb]) (logid:61204b11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:14,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e6cd0a]) (logid:ffd53921) No inactive management server node found 2025-06-29 03:33:14,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e6cd0a]) (logid:ffd53921) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:33:14,380 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-55591d65]) (logid:82b3a14e) Starting to remove all vm_stats rows older than [Sat Jun 28 15:33:14 CEST 2025]. 2025-06-29 03:33:14,405 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-55591d65]) (logid:82b3a14e) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:33:14 CEST 2025]. 2025-06-29 03:33:14,449 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-bd7ae38c]) (logid:d466f001) HA health check task is running... 2025-06-29 03:33:15,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85e31a09]) (logid:b629cdd4) No inactive management server node found 2025-06-29 03:33:15,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85e31a09]) (logid:b629cdd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:17,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38c318a0]) (logid:ff8c6f38) No inactive management server node found 2025-06-29 03:33:17,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38c318a0]) (logid:ff8c6f38) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:18,450 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c75b4e8f]) (logid:f034dac4) HA health check task is running... 2025-06-29 03:33:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b87b574]) (logid:90cfebd5) No inactive management server node found 2025-06-29 03:33:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b87b574]) (logid:90cfebd5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:20,195 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8218: Seq 7-8218: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:33:20,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8218 2025-06-29 03:33:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0352084b]) (logid:eb1e2f7c) No inactive management server node found 2025-06-29 03:33:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0352084b]) (logid:eb1e2f7c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:21,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b6072a6]) (logid:f5c1bfe3) No inactive management server node found 2025-06-29 03:33:21,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b6072a6]) (logid:f5c1bfe3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:22,451 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-437a64ae]) (logid:08688e02) HA health check task is running... 2025-06-29 03:33:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cbb44f3]) (logid:63113227) No inactive management server node found 2025-06-29 03:33:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cbb44f3]) (logid:63113227) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:24,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-354a26c2]) (logid:5d6fc3ef) No inactive management server node found 2025-06-29 03:33:24,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-354a26c2]) (logid:5d6fc3ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:25,512 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) StorageCollector is running... 2025-06-29 03:33:25,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:33:25,518 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:33:25,518 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:33:25,519 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-6717118844223095508: Routed from 90520738888109 2025-06-29 03:33:25,574 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 4-6717118844223095508: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:33:25,579 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:33:25,579 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:33:25,579 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:33:25,580 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-3048092522799696808: Routed from 90520738888109 2025-06-29 03:33:25,628 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 1-3048092522799696808: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:33:25,638 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:33:25,638 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:33:25,638 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:33:25,638 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-1612007191621930092: Routed from 90520738888109 2025-06-29 03:33:25,650 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-1612007191621930092: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:33:25,662 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:33:25,662 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:33:25,662 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:33:25,663 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-3048092522799696809: Routed from 90520738888109 2025-06-29 03:33:26,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f040704]) (logid:337dde04) No inactive management server node found 2025-06-29 03:33:26,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f040704]) (logid:337dde04) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:33:26,452 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-49c07606]) (logid:da943fb5) HA health check task is running... 2025-06-29 03:33:26,754 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 1-3048092522799696809: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:33:26,756 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:33:26,756 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:33:26,756 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:33:26,756 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-1612007191621930093: Routed from 90520738888109 2025-06-29 03:33:26,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba9d1525]) (logid:262a70e5) Enabled non-edge zones available for scan: 1 2025-06-29 03:33:26,764 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ba9d1525]) (logid:262a70e5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:33:26,766 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba9d1525]) (logid:262a70e5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:33:26,894 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d124c47c]) (logid:bf647659) Enabled non-edge zones available for scan: 1 2025-06-29 03:33:26,898 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d124c47c]) (logid:bf647659) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:33:27,815 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45bf80b7]) (logid:b38f6442) Seq 2-1612007191621930093: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:33:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36c59fc6]) (logid:1c94dbf3) No inactive management server node found 2025-06-29 03:33:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36c59fc6]) (logid:1c94dbf3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f1ad35c]) (logid:706fa495) No inactive management server node found 2025-06-29 03:33:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f1ad35c]) (logid:706fa495) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:30,194 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8219: Seq 7-8219: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:33:30,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8219 2025-06-29 03:33:30,454 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ffd42983]) (logid:18c6e931) HA health check task is running... 2025-06-29 03:33:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-732baeeb]) (logid:6f735f40) No inactive management server node found 2025-06-29 03:33:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-732baeeb]) (logid:6f735f40) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:31,068 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:33:31,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:33:31,069 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) Seq 2-3048092522799696810: Routed from 90520738888109 2025-06-29 03:33:31,072 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) Seq 1-3048092522799696810: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:33:31,073 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:33:31,073 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) Seq 2-1612007191621930094: Routed from 90520738888109 2025-06-29 03:33:31,076 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-24b28f10]) (logid:8ac573cb) Seq 2-1612007191621930094: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:33:32,340 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8500f1ff]) (logid:9d081a61) No inactive management server node found 2025-06-29 03:33:32,340 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8500f1ff]) (logid:9d081a61) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:33,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-252c7b98]) (logid:2942c0e0) No inactive management server node found 2025-06-29 03:33:33,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-252c7b98]) (logid:2942c0e0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:33,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:33:33,980 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:33:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:33:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:33:34,455 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c96d142c]) (logid:cd3af2e8) HA health check task is running... 2025-06-29 03:33:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-606571b3]) (logid:7e4c34c5) No inactive management server node found 2025-06-29 03:33:35,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-606571b3]) (logid:7e4c34c5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:36,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3510c64c]) (logid:fb9db1eb) No inactive management server node found 2025-06-29 03:33:36,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3510c64c]) (logid:fb9db1eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5f15c6f]) (logid:be29eb60) No inactive management server node found 2025-06-29 03:33:38,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5f15c6f]) (logid:be29eb60) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:38,456 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-908429eb]) (logid:c4597245) HA health check task is running... 2025-06-29 03:33:39,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55836de2]) (logid:30ec41cc) No inactive management server node found 2025-06-29 03:33:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55836de2]) (logid:30ec41cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:40,194 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8220: Seq 7-8220: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:33:40,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8220 2025-06-29 03:33:40,817 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-eb84ea85]) (logid:5bcdadff) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:33:40,818 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-eb84ea85]) (logid:5bcdadff) Snapshot scheduler is being called at [2025-06-29T01:33:40+0000]. 2025-06-29 03:33:40,819 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-eb84ea85]) (logid:5bcdadff) There are [0] scheduled snapshots to be executed at [2025-06-29T01:33:40+0000]. 2025-06-29 03:33:40,917 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-2fe83428]) (logid:c45cc8f5) External devices are not used. Skipping external device usage collection 2025-06-29 03:33:40,980 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-82996c3d]) (logid:c47f844f) Backup backup.poll is being called at 2025-06-29T01:33:40+0000 2025-06-29 03:33:40,998 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-8d0d19c3]) (logid:2319afd9) Found 0 running routers. 2025-06-29 03:33:40,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f308c156]) (logid:10076ca9) Found 0 routers to update status. 2025-06-29 03:33:41,001 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f308c156]) (logid:10076ca9) Found 0 VPC's to update Redundant State. 2025-06-29 03:33:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f308c156]) (logid:10076ca9) Found 0 networks to update RvR status. 2025-06-29 03:33:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-42c27335]) (logid:b9dd5f5b) DbCollector is running... 2025-06-29 03:33:41,289 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-754f6fb6]) (logid:100cd28b) Started resource counters recalculation periodic task. 2025-06-29 03:33:41,293 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-754f6fb6]) (logid:100cd28b) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:33:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ac012ed]) (logid:77cfa962) No inactive management server node found 2025-06-29 03:33:41,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ac012ed]) (logid:77cfa962) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:41,649 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-754f6fb6]) (logid:100cd28b) Finished resource counters recalculation periodic task. 2025-06-29 03:33:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) ManagementServerCollector is running... 2025-06-29 03:33:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) DB properties were already loaded 2025-06-29 03:33:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:33:42,168 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Execution of process [4064055] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:33:42,169 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Exit value of process [4064055] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:33:42,169 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Process [4064055] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:33:42,169 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) usage install: null 2025-06-29 03:33:42,169 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Sessions active: 0 2025-06-29 03:33:42,171 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:33:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064058] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:33:42,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:33:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064063] for command [/bin/bash -c uname -r ]. 2025-06-29 03:33:42,182 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:33:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064064] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:33:42,189 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) system memory from /proc: 4887515136 2025-06-29 03:33:42,189 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:33:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064069] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:33:42,195 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) free memory from /proc: 494579712 2025-06-29 03:33:42,195 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:33:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064074] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:33:42,208 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) used memory from /proc: 769996 2025-06-29 03:33:42,208 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:33:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064075] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:33:42,216 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:33:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064076] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:33:42,234 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:33:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064077] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:33:42,242 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:33:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064084] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:33:42,248 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:33:42,254 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Successfully executed process [4064085] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail buffersdirect.count, 120 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail buffersdirect.used, 658750 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail buffersmapped.count, 7 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail buffersmapped.used, 798602 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail gcPS-Scavenge.count, 525 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail gcPS-Scavenge.time, 3801 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail jvmuptime, 44762333 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memoryheap.committed, 359137280 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memoryheap.init, 77594624 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memoryheap.usage, 0.11401664316899761 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorynon-heap.max, -1 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorynon-heap.usage, -2.20751752E8 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorynon-heap.used, 220751752 2025-06-29 03:33:42,256 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765383397760597 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30359166194554604 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.Metaspace.usage, 0.9859345237273631 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.PS-Eden-Space.usage, 0.03954635539015044 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.PS-Old-Gen.usage, 0.12689857664495893 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorypools.PS-Survivor-Space.usage, 0.6901086171468099 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorytotal.committed, 584253440 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorytotal.init, 85262336 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorytotal.max, 1908932607 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail memorytotal.used, 438463496 2025-06-29 03:33:42,257 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail threadsdeadlocks, [] 2025-06-29 03:33:42,258 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail threadsnew.count, 0 2025-06-29 03:33:42,259 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:33:42,260 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-4be20802]) (logid:4dabbef8) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:33:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142238808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44762246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":437650224,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217650088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494579712,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.26,0.16],"systemCyclesUsage":[66378451,53817885,4588589786],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:33:42,260 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1491, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:33:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142238808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44762246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":437650224,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217650088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494579712,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.26,0.16],"systemCyclesUsage":[66378451,53817885,4588589786],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:33:42,261 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:33:42,265 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:33:42,265 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-f84d5f66]) (logid:d1ef00be) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1491, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:33:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142238808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44762246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":437650224,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217650088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494579712,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.26,0.16],"systemCyclesUsage":[66378451,53817885,4588589786],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:33:42,265 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-335:[ctx-aa84b5e1]) (logid:72e390ec) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:33:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142238808,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44762246,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.2,"totalInit":85262336,"totalUsed":437650224,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217650088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":217,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":494579712,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.2,0.26,0.16],"systemCyclesUsage":[66378451,53817885,4588589786],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:33:42,457 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4dfccb85]) (logid:c25f89eb) HA health check task is running... 2025-06-29 03:33:42,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e0dd205]) (logid:389c5c78) No inactive management server node found 2025-06-29 03:33:42,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e0dd205]) (logid:389c5c78) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:44,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-388bc61e]) (logid:8036f3c3) No inactive management server node found 2025-06-29 03:33:44,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-388bc61e]) (logid:8036f3c3) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:33:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:33:45,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49efbd84]) (logid:b697b1eb) No inactive management server node found 2025-06-29 03:33:45,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49efbd84]) (logid:b697b1eb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:46,047 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ab912d42]) (logid:3b315bf5) VM scheduler.poll is being called at 2025-06-29T01:34:00+0000 2025-06-29 03:33:46,050 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ab912d42]) (logid:3b315bf5) Got 0 scheduled jobs to be executed at 2025-06-29T01:34:00+0000 2025-06-29 03:33:46,051 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ab912d42]) (logid:3b315bf5) Cleaned up 0 VM scheduled job entries 2025-06-29 03:33:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-6f8c2dd1]) (logid:2330f418) VM Operation Thread Running 2025-06-29 03:33:46,192 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-1:[ctx-e5803e5e]) (logid:8e90ced9) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:33:46,333 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-14eb7092]) (logid:894109f7) ClusterDRS.poll is being called at 2025-06-29T01:34:00+0000 2025-06-29 03:33:46,340 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-14eb7092]) (logid:894109f7) Removed 0 old drs migration plans 2025-06-29 03:33:46,459 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8e877730]) (logid:d3c71d91) HA health check task is running... 2025-06-29 03:33:47,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:33:47,240 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:33:47,241 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:33:47,244 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:33:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4218082]) (logid:ccaf9ad8) No inactive management server node found 2025-06-29 03:33:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4218082]) (logid:ccaf9ad8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:48,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fe4d5f2]) (logid:0cf3bef1) No inactive management server node found 2025-06-29 03:33:48,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fe4d5f2]) (logid:0cf3bef1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:50,195 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8222: Seq 7-8222: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:33:50,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8222 2025-06-29 03:33:50,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1118512d]) (logid:ef2ced91) No inactive management server node found 2025-06-29 03:33:50,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1118512d]) (logid:ef2ced91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:50,460 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c7b1dc06]) (logid:d008c88e) HA health check task is running... 2025-06-29 03:33:51,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44f87679]) (logid:cef7c9ff) No inactive management server node found 2025-06-29 03:33:51,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44f87679]) (logid:cef7c9ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:53,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-836c6a3c]) (logid:6eacbe4d) No inactive management server node found 2025-06-29 03:33:53,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-836c6a3c]) (logid:6eacbe4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:54,461 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-61c22c20]) (logid:205433e7) HA health check task is running... 2025-06-29 03:33:54,830 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-916a34d1]) (logid:8e97433a) No inactive management server node found 2025-06-29 03:33:54,830 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-916a34d1]) (logid:8e97433a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:56,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9baf2e4f]) (logid:dfc08f93) No inactive management server node found 2025-06-29 03:33:56,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9baf2e4f]) (logid:dfc08f93) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:33:56,757 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0ef0baff]) (logid:046fb1f7) Enabled non-edge zones available for scan: 1 2025-06-29 03:33:56,760 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0ef0baff]) (logid:046fb1f7) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:33:56,761 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0ef0baff]) (logid:046fb1f7) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:33:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-454fc8ca]) (logid:9cdec778) Enabled non-edge zones available for scan: 1 2025-06-29 03:33:56,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-454fc8ca]) (logid:9cdec778) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:33:57,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c6d82b]) (logid:437f4503) No inactive management server node found 2025-06-29 03:33:57,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c6d82b]) (logid:437f4503) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:33:57,972 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-c2151967]) (logid:25b58b1a) AutoScaling Monitor is running... 2025-06-29 03:33:58,102 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-a837f33e]) (logid:372b0f3e) Starting to remove all volume_stats rows older than [Sat Jun 28 15:33:58 CEST 2025]. 2025-06-29 03:33:58,103 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-a837f33e]) (logid:372b0f3e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:33:58 CEST 2025]. 2025-06-29 03:33:58,462 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1f223e3a]) (logid:fff19012) HA health check task is running... 2025-06-29 03:33:59,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a977a686]) (logid:4557c3e6) No inactive management server node found 2025-06-29 03:33:59,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a977a686]) (logid:4557c3e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:00,195 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8223: Seq 7-8223: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:00,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8223 2025-06-29 03:34:00,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1acf1d3e]) (logid:516f2593) No inactive management server node found 2025-06-29 03:34:00,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1acf1d3e]) (logid:516f2593) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:02,337 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6144e3b]) (logid:9c2eb3da) No inactive management server node found 2025-06-29 03:34:02,337 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6144e3b]) (logid:9c2eb3da) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 3ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:02,463 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d65afe46]) (logid:f2f7d296) HA health check task is running... 2025-06-29 03:34:03,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5c6e665]) (logid:b93e9c1e) No inactive management server node found 2025-06-29 03:34:03,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5c6e665]) (logid:b93e9c1e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:05,339 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd851321]) (logid:d58b6f78) No inactive management server node found 2025-06-29 03:34:05,339 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd851321]) (logid:d58b6f78) Peer scan is finished. profiler: Done. Duration: 8ms , profilerQueryActiveList: Done. Duration: 5ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:34:06,464 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d66dbd63]) (logid:3e8318ce) HA health check task is running... 2025-06-29 03:34:06,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ea6c258]) (logid:eebb64ff) No inactive management server node found 2025-06-29 03:34:06,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ea6c258]) (logid:eebb64ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:07,766 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:34:07,768 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:34:07,769 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) Seq 2-3048092522799696811: Routed from 90520738888109 2025-06-29 03:34:07,821 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) Seq 1-3048092522799696811: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:34:07,859 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:34:07,860 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) Seq 2-1612007191621930095: Routed from 90520738888109 2025-06-29 03:34:07,911 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-01ee7b39]) (logid:274518f9) Seq 2-1612007191621930095: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:34:08,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b8a6ba0]) (logid:8f8b8473) No inactive management server node found 2025-06-29 03:34:08,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b8a6ba0]) (logid:8f8b8473) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:09,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83049b14]) (logid:b9c24056) No inactive management server node found 2025-06-29 03:34:09,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83049b14]) (logid:b9c24056) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:10,196 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8224: Seq 7-8224: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:10,218 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8224 2025-06-29 03:34:10,466 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b46b5ab5]) (logid:3902089b) HA health check task is running... 2025-06-29 03:34:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cca52b9f]) (logid:62b7be6c) Found 0 routers to update status. 2025-06-29 03:34:11,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cca52b9f]) (logid:62b7be6c) Found 0 VPC's to update Redundant State. 2025-06-29 03:34:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cca52b9f]) (logid:62b7be6c) Found 0 networks to update RvR status. 2025-06-29 03:34:11,068 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Running Capacity Checker ... 2025-06-29 03:34:11,069 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) recalculating system capacity 2025-06-29 03:34:11,069 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Executing cpu/ram capacity update 2025-06-29 03:34:11,073 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:11,074 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:11,082 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:11,087 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:34:11,087 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:34:11,087 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:34:11,109 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:11,110 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:11,115 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:11,116 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:34:11,117 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:34:11,117 DEBUG [c.c.c.CapacityManagerImpl] (pool-326-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:34:11,126 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Done executing cpu/ram capacity update 2025-06-29 03:34:11,126 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Executing storage capacity update 2025-06-29 03:34:11,134 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:34:11,134 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:34:11,134 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:34:11,137 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:34:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:34:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:34:11,143 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:34:11,145 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:34:11,151 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:34:11,152 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:34:11,152 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:34:11,160 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:34:11,165 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:34:11,166 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:34:11,166 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:34:11,169 DEBUG [c.c.s.StorageManagerImpl] (pool-327-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:34:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Done executing storage capacity update 2025-06-29 03:34:11,170 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Executing capacity updates for public ip and Vlans 2025-06-29 03:34:11,186 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Done capacity updates for public ip and Vlans 2025-06-29 03:34:11,186 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Executing capacity updates for private ip 2025-06-29 03:34:11,190 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Done executing capacity updates for private ip 2025-06-29 03:34:11,190 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Done recalculating system capacity 2025-06-29 03:34:11,218 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5de13165]) (logid:2d2e924a) Done running Capacity Checker ... 2025-06-29 03:34:11,261 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-de82fa42]) (logid:becec352) Checking if any host reservation can be released ... 2025-06-29 03:34:11,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-de82fa42]) (logid:becec352) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:11,272 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-de82fa42]) (logid:becec352) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:11,272 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-de82fa42]) (logid:becec352) Done running HostReservationReleaseChecker ... 2025-06-29 03:34:11,324 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dfd584d]) (logid:8c45f0ae) No inactive management server node found 2025-06-29 03:34:11,324 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dfd584d]) (logid:8c45f0ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:34:12,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f37c437]) (logid:82824a96) No inactive management server node found 2025-06-29 03:34:12,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f37c437]) (logid:82824a96) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:14,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d9a7c71]) (logid:b39ce8c4) No inactive management server node found 2025-06-29 03:34:14,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d9a7c71]) (logid:b39ce8c4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:14,407 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-b1be1a1f]) (logid:52774705) Starting to remove all vm_stats rows older than [Sat Jun 28 15:34:14 CEST 2025]. 2025-06-29 03:34:14,431 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-b1be1a1f]) (logid:52774705) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:34:14 CEST 2025]. 2025-06-29 03:34:14,467 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3a3cca12]) (logid:13278020) HA health check task is running... 2025-06-29 03:34:15,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42569e3c]) (logid:d136a0a7) No inactive management server node found 2025-06-29 03:34:15,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42569e3c]) (logid:d136a0a7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:17,056 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bf8eb8cc]) (logid:5671c246) Resetting hosts suitable for reconnect 2025-06-29 03:34:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bf8eb8cc]) (logid:5671c246) Completed resetting hosts suitable for reconnect 2025-06-29 03:34:17,058 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bf8eb8cc]) (logid:5671c246) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:34:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bf8eb8cc]) (logid:5671c246) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:34:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bf8eb8cc]) (logid:5671c246) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:34:17,059 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-bf8eb8cc]) (logid:5671c246) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:34:17,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e932985b]) (logid:8ae56bbc) No inactive management server node found 2025-06-29 03:34:17,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e932985b]) (logid:8ae56bbc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:18,468 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-662873bf]) (logid:7176f818) HA health check task is running... 2025-06-29 03:34:18,834 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2da78f9d]) (logid:53812e44) No inactive management server node found 2025-06-29 03:34:18,834 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2da78f9d]) (logid:53812e44) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:20,196 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8225: Seq 7-8225: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:20,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8225 2025-06-29 03:34:20,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f04350d8]) (logid:118c3299) No inactive management server node found 2025-06-29 03:34:20,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f04350d8]) (logid:118c3299) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3597fc9]) (logid:b708f255) No inactive management server node found 2025-06-29 03:34:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3597fc9]) (logid:b708f255) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:22,470 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f137e3c1]) (logid:3d2cbaf1) HA health check task is running... 2025-06-29 03:34:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e649cb0e]) (logid:97054246) No inactive management server node found 2025-06-29 03:34:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e649cb0e]) (logid:97054246) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:24,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80dce88f]) (logid:4c758bf6) No inactive management server node found 2025-06-29 03:34:24,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80dce88f]) (logid:4c758bf6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:25,196 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8226: Seq 7-8226: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:25,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8226 2025-06-29 03:34:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-611f9bc8]) (logid:5a264646) No inactive management server node found 2025-06-29 03:34:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-611f9bc8]) (logid:5a264646) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:26,471 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-04fab5e6]) (logid:504a99cf) HA health check task is running... 2025-06-29 03:34:26,758 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-59570afc]) (logid:d41ed64e) Enabled non-edge zones available for scan: 1 2025-06-29 03:34:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-59570afc]) (logid:d41ed64e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:34:26,764 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-59570afc]) (logid:d41ed64e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:34:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3940d599]) (logid:da383400) Enabled non-edge zones available for scan: 1 2025-06-29 03:34:26,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3940d599]) (logid:da383400) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:34:27,815 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) StorageCollector is running... 2025-06-29 03:34:27,823 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:34:27,823 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:34:27,823 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:34:27,823 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 2-6717118844223095509: Routed from 90520738888109 2025-06-29 03:34:27,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4838ac9d]) (logid:eccf9fcf) No inactive management server node found 2025-06-29 03:34:27,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4838ac9d]) (logid:eccf9fcf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:27,898 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 4-6717118844223095509: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:34:27,902 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:34:27,902 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:34:27,902 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:34:27,903 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 2-3048092522799696812: Routed from 90520738888109 2025-06-29 03:34:27,950 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 1-3048092522799696812: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:34:27,952 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:34:27,952 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:34:27,952 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:34:27,953 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 2-3048092522799696813: Routed from 90520738888109 2025-06-29 03:34:28,042 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 1-3048092522799696813: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:34:28,054 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:34:28,054 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:34:28,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:34:28,055 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 2-3048092522799696814: Routed from 90520738888109 2025-06-29 03:34:29,146 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 1-3048092522799696814: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:34:29,148 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:34:29,148 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:34:29,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:34:29,149 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 2-1612007191621930096: Routed from 90520738888109 2025-06-29 03:34:29,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be42ac0f]) (logid:2429ff3f) No inactive management server node found 2025-06-29 03:34:29,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be42ac0f]) (logid:2429ff3f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:30,171 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f2d50ff9]) (logid:1d1e1af1) Seq 2-1612007191621930096: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:34:30,472 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a930b556]) (logid:67d78f63) HA health check task is running... 2025-06-29 03:34:30,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dc9333e]) (logid:63bf6904) No inactive management server node found 2025-06-29 03:34:30,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dc9333e]) (logid:63bf6904) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:31,079 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:34:31,079 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:34:31,080 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) Seq 2-3048092522799696815: Routed from 90520738888109 2025-06-29 03:34:31,082 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) Seq 1-3048092522799696815: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:34:31,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:34:31,084 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) Seq 2-1612007191621930097: Routed from 90520738888109 2025-06-29 03:34:31,087 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-cd0efb42]) (logid:467d40b6) Seq 2-1612007191621930097: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:34:32,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1292f0e]) (logid:ce3c2e68) No inactive management server node found 2025-06-29 03:34:32,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1292f0e]) (logid:ce3c2e68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:33,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9498497c]) (logid:2aa6c5d5) No inactive management server node found 2025-06-29 03:34:33,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9498497c]) (logid:2aa6c5d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:33,979 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:33,981 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:33,983 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:34:33,986 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:34:34,474 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-14d25a88]) (logid:152424c7) HA health check task is running... 2025-06-29 03:34:35,197 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8227: Seq 7-8227: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:35,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8227 2025-06-29 03:34:35,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41c36124]) (logid:90f9ff92) No inactive management server node found 2025-06-29 03:34:35,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41c36124]) (logid:90f9ff92) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:36,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e5ea8bf]) (logid:3ed3d15b) No inactive management server node found 2025-06-29 03:34:36,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e5ea8bf]) (logid:3ed3d15b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:38,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a25ffe7]) (logid:dbf0929e) No inactive management server node found 2025-06-29 03:34:38,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a25ffe7]) (logid:dbf0929e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:38,475 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d74ad003]) (logid:bac70d60) HA health check task is running... 2025-06-29 03:34:39,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7be34eed]) (logid:54b10479) No inactive management server node found 2025-06-29 03:34:39,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7be34eed]) (logid:54b10479) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:41,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cfb389bb]) (logid:8af145a9) Found 0 routers to update status. 2025-06-29 03:34:41,002 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cfb389bb]) (logid:8af145a9) Found 0 VPC's to update Redundant State. 2025-06-29 03:34:41,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cfb389bb]) (logid:8af145a9) Found 0 networks to update RvR status. 2025-06-29 03:34:41,155 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-90b2d263]) (logid:0c2c8a29) DbCollector is running... 2025-06-29 03:34:41,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e5c61f1]) (logid:d90059f8) No inactive management server node found 2025-06-29 03:34:41,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e5c61f1]) (logid:d90059f8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:34:42,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) ManagementServerCollector is running... 2025-06-29 03:34:42,153 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) DB properties were already loaded 2025-06-29 03:34:42,153 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:34:42,168 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Execution of process [4064103] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:34:42,168 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Exit value of process [4064103] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:34:42,168 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Process [4064103] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:34:42,168 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) usage install: null 2025-06-29 03:34:42,168 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Sessions active: 0 2025-06-29 03:34:42,170 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:34:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064106] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:34:42,176 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:34:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064111] for command [/bin/bash -c uname -r ]. 2025-06-29 03:34:42,179 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:34:42,185 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064112] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:34:42,186 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) system memory from /proc: 4887515136 2025-06-29 03:34:42,186 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:34:42,191 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064117] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:34:42,192 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) free memory from /proc: 493846528 2025-06-29 03:34:42,192 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:34:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064122] for command [/bin/bash -c ps -o rss= 4028106 ]. 2025-06-29 03:34:42,204 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) used memory from /proc: 769996 2025-06-29 03:34:42,204 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:34:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064123] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:34:42,209 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:34:42,221 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064124] for command [/bin/bash -c ps -o vsz= 4028106 ]. 2025-06-29 03:34:42,221 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:34:42,228 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064125] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:34:42,229 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:34:42,232 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064132] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:34:42,233 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:34:42,238 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Successfully executed process [4064133] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail buffersdirect.capacity, 658750 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail buffersdirect.count, 120 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail buffersdirect.used, 658750 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail buffersmapped.capacity, 798602 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail buffersmapped.count, 7 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail buffersmapped.used, 798602 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail gcPS-MarkSweep.count, 7 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail gcPS-MarkSweep.time, 1436 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail gcPS-Scavenge.count, 526 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail gcPS-Scavenge.time, 3807 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail jvmname, 4028106@cs-mgmt1.cloud.redacted.net 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail jvmuptime, 44822317 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memoryheap.committed, 359137280 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memoryheap.init, 77594624 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memoryheap.usage, 0.11373390924861816 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorynon-heap.committed, 225116160 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorynon-heap.max, -1 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorynon-heap.usage, -2.20751752E8 2025-06-29 03:34:42,240 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorynon-heap.used, 220751752 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30835526315789474 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1765383397760597 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.30359166194554604 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.Compressed-Class-Space.usage, 0.012714900076389313 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.Metaspace.usage, 0.9859345237273631 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.PS-Eden-Space.usage, 0.0371402636125095 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.PS-Old-Gen.usage, 0.1269157406874342 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorypools.PS-Survivor-Space.usage, 0.7786553700764974 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorytotal.committed, 584253440 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorytotal.init, 85262336 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorytotal.max, 1908932607 2025-06-29 03:34:42,241 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail memorytotal.used, 437919880 2025-06-29 03:34:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail threadsdeadlocks, [] 2025-06-29 03:34:42,242 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail threadsnew.count, 0 2025-06-29 03:34:42,243 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) not storing detail threadstimed_waiting.count, 85 2025-06-29 03:34:42,244 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-4019338c]) (logid:0bd69b85) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:34:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142788768,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":437100264,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217110368,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493846528,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.21,0.15],"systemCyclesUsage":[66378760,53818195,4588612903],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:34:42,245 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1493, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:34:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142788768,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":437100264,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217110368,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493846528,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.21,0.15],"systemCyclesUsage":[66378760,53818195,4588612903],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:34:42,245 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:34:42,249 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-335:[ctx-89f885cd]) (logid:4f28ecb0) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:34:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142788768,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":437100264,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217110368,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493846528,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.21,0.15],"systemCyclesUsage":[66378760,53818195,4588612903],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:34:42,249 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 03:34:42,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-3c87c39f]) (logid:afd065fb) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 1493, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751116087255,"collectionTime":"Jun 29, 2025, 3:34:42 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":359137280,"freeJvmMemoryBytes":142788768,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":44822245,"jvmStartTime":1751116059961,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":437100264,"totalCommitted":584253440,"pid":4028106,"jvmName":"4028106@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":217110368,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":124,"systemMemoryTotal":4887515136,"systemMemoryFree":493846528,"systemMemoryUsed":769996,"systemMemoryVirtualSize":6306000896,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.21,0.15],"systemCyclesUsage":[66378760,53818195,4588612903],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:34:42,476 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-32fa203d]) (logid:ecf72b7e) HA health check task is running... 2025-06-29 03:34:42,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96af0aa1]) (logid:b24bd914) No inactive management server node found 2025-06-29 03:34:42,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96af0aa1]) (logid:b24bd914) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:44,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46e50dcd]) (logid:e6683580) No inactive management server node found 2025-06-29 03:34:44,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46e50dcd]) (logid:e6683580) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:45,197 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8228: Seq 7-8228: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:45,218 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8228 2025-06-29 03:34:45,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:34:45,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:34:45,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76b3d936]) (logid:09be1552) No inactive management server node found 2025-06-29 03:34:45,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76b3d936]) (logid:09be1552) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:46,048 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-470f311c]) (logid:d7e3a850) VM scheduler.poll is being called at 2025-06-29T01:35:00+0000 2025-06-29 03:34:46,051 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-470f311c]) (logid:d7e3a850) Got 0 scheduled jobs to be executed at 2025-06-29T01:35:00+0000 2025-06-29 03:34:46,053 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-470f311c]) (logid:d7e3a850) Cleaned up 0 VM scheduled job entries 2025-06-29 03:34:46,057 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-7dbd6d4a]) (logid:05a3bc1a) VM Operation Thread Running 2025-06-29 03:34:46,334 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f192492d]) (logid:31b0ac0d) ClusterDRS.poll is being called at 2025-06-29T01:35:00+0000 2025-06-29 03:34:46,343 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f192492d]) (logid:31b0ac0d) Removed 0 old drs migration plans 2025-06-29 03:34:46,477 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6070e997]) (logid:51ed645c) HA health check task is running... 2025-06-29 03:34:47,234 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:47,236 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:47,238 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:34:47,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:34:47,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2aa6a7b5]) (logid:03638987) No inactive management server node found 2025-06-29 03:34:47,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2aa6a7b5]) (logid:03638987) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:48,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f21ec657]) (logid:58b54ecf) No inactive management server node found 2025-06-29 03:34:48,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f21ec657]) (logid:58b54ecf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:50,335 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b969b9]) (logid:01da8ae9) No inactive management server node found 2025-06-29 03:34:50,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b969b9]) (logid:01da8ae9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:50,478 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ce68676c]) (logid:328f8ce3) HA health check task is running... 2025-06-29 03:34:51,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9fc7156]) (logid:96b1e6c9) No inactive management server node found 2025-06-29 03:34:51,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9fc7156]) (logid:96b1e6c9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:53,341 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-356f8663]) (logid:ed0e3112) No inactive management server node found 2025-06-29 03:34:53,341 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-356f8663]) (logid:ed0e3112) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 5ms 2025-06-29 03:34:54,479 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1ea0b7fe]) (logid:a2f2d99b) HA health check task is running... 2025-06-29 03:34:54,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4ab99c2]) (logid:89eb1f91) No inactive management server node found 2025-06-29 03:34:54,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4ab99c2]) (logid:89eb1f91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:55,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8230: Seq 7-8230: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:34:55,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8230 2025-06-29 03:34:56,332 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29082c7b]) (logid:798395e9) No inactive management server node found 2025-06-29 03:34:56,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29082c7b]) (logid:798395e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:56,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-422ed263]) (logid:e63e9b97) Enabled non-edge zones available for scan: 1 2025-06-29 03:34:56,759 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-422ed263]) (logid:e63e9b97) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:34:56,760 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-422ed263]) (logid:e63e9b97) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:34:56,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8cdd9c34]) (logid:fdccdfc1) Enabled non-edge zones available for scan: 1 2025-06-29 03:34:56,899 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8cdd9c34]) (logid:fdccdfc1) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:34:57,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4334de8]) (logid:b0a62073) No inactive management server node found 2025-06-29 03:34:57,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4334de8]) (logid:b0a62073) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:34:57,974 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-bc1aeb6b]) (logid:ab1c0024) AutoScaling Monitor is running... 2025-06-29 03:34:58,105 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-17170b63]) (logid:a5c41987) Starting to remove all volume_stats rows older than [Sat Jun 28 15:34:58 CEST 2025]. 2025-06-29 03:34:58,105 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-17170b63]) (logid:a5c41987) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:34:58 CEST 2025]. 2025-06-29 03:34:58,480 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5698573c]) (logid:20069f36) HA health check task is running... 2025-06-29 03:34:59,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dacb90f]) (logid:2c662dce) No inactive management server node found 2025-06-29 03:34:59,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dacb90f]) (logid:2c662dce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:00,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84ed9498]) (logid:527699e0) No inactive management server node found 2025-06-29 03:35:00,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84ed9498]) (logid:527699e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:02,330 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:35:02,355 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfbf3cf6]) (logid:caabeaa1) No inactive management server node found 2025-06-29 03:35:02,355 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfbf3cf6]) (logid:caabeaa1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:02,482 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-36d19dbe]) (logid:219bf80c) HA health check task is running... 2025-06-29 03:35:03,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6aa88aec]) (logid:940be2a8) No inactive management server node found 2025-06-29 03:35:03,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6aa88aec]) (logid:940be2a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:05,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8231: Seq 7-8231: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:35:05,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8231 2025-06-29 03:35:05,323 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be41a657]) (logid:375d3e64) No inactive management server node found 2025-06-29 03:35:05,323 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be41a657]) (logid:375d3e64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:06,483 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-eee869c3]) (logid:d0752411) HA health check task is running... 2025-06-29 03:35:06,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-058709bc]) (logid:5923dfe5) No inactive management server node found 2025-06-29 03:35:06,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-058709bc]) (logid:5923dfe5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:07,967 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:35:07,970 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:35:07,971 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) Seq 2-3048092522799696816: Routed from 90520738888109 2025-06-29 03:35:08,023 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) Seq 1-3048092522799696816: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:35:08,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:35:08,058 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) Seq 2-1612007191621930098: Routed from 90520738888109 2025-06-29 03:35:08,114 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-de989664]) (logid:4df16bab) Seq 2-1612007191621930098: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:35:08,321 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5890062]) (logid:da28c4d0) No inactive management server node found 2025-06-29 03:35:08,321 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5890062]) (logid:da28c4d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:09,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f949404b]) (logid:731dbcfb) No inactive management server node found 2025-06-29 03:35:09,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f949404b]) (logid:731dbcfb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:10,484 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-82dcec0b]) (logid:8550fccd) HA health check task is running... 2025-06-29 03:35:10,999 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-17cbde72]) (logid:a2472c48) Found 0 routers to update status. 2025-06-29 03:35:11,000 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-17cbde72]) (logid:a2472c48) Found 0 VPC's to update Redundant State. 2025-06-29 03:35:11,003 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-17cbde72]) (logid:a2472c48) Found 0 networks to update RvR status. 2025-06-29 03:35:11,322 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b3e3a6e]) (logid:0777abc6) No inactive management server node found 2025-06-29 03:35:11,322 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b3e3a6e]) (logid:0777abc6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:12,824 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f89608ee]) (logid:dca78bc4) No inactive management server node found 2025-06-29 03:35:12,824 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f89608ee]) (logid:dca78bc4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:14,336 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e41bec74]) (logid:782b5b80) No inactive management server node found 2025-06-29 03:35:14,336 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e41bec74]) (logid:782b5b80) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:14,432 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-205245bb]) (logid:cb730d65) Starting to remove all vm_stats rows older than [Sat Jun 28 15:35:14 CEST 2025]. 2025-06-29 03:35:14,481 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-205245bb]) (logid:cb730d65) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:35:14 CEST 2025]. 2025-06-29 03:35:14,486 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5831594b]) (logid:e543b5c2) HA health check task is running... 2025-06-29 03:35:15,199 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8232: Seq 7-8232: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:35:15,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8232 2025-06-29 03:35:15,823 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1450dcec]) (logid:ed971f6b) No inactive management server node found 2025-06-29 03:35:15,823 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1450dcec]) (logid:ed971f6b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:17,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d66e47b]) (logid:f3520d4f) No inactive management server node found 2025-06-29 03:35:17,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d66e47b]) (logid:f3520d4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:18,487 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-39b4838f]) (logid:1f168adb) HA health check task is running... 2025-06-29 03:35:18,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-677b1e85]) (logid:eb0e8c9b) No inactive management server node found 2025-06-29 03:35:18,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-677b1e85]) (logid:eb0e8c9b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:20,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e276c90]) (logid:5371c363) No inactive management server node found 2025-06-29 03:35:20,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e276c90]) (logid:5371c363) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:21,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b523f4f3]) (logid:c133aebd) No inactive management server node found 2025-06-29 03:35:21,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b523f4f3]) (logid:c133aebd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:22,489 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c5bb3a73]) (logid:e1858a1b) HA health check task is running... 2025-06-29 03:35:23,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-949fccff]) (logid:2aedc4aa) No inactive management server node found 2025-06-29 03:35:23,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-949fccff]) (logid:2aedc4aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:35:24,832 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c9ca9e5]) (logid:63b72c7e) No inactive management server node found 2025-06-29 03:35:24,832 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c9ca9e5]) (logid:63b72c7e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:25,199 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8233: Seq 7-8233: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:35:25,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8233 2025-06-29 03:35:26,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3663dbca]) (logid:aa0c2104) No inactive management server node found 2025-06-29 03:35:26,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3663dbca]) (logid:aa0c2104) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:26,491 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d8595e4a]) (logid:3e6decc6) HA health check task is running... 2025-06-29 03:35:26,756 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba1ac283]) (logid:8a14d929) Enabled non-edge zones available for scan: 1 2025-06-29 03:35:26,762 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ba1ac283]) (logid:8a14d929) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:35:26,762 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba1ac283]) (logid:8a14d929) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:35:26,895 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3441852d]) (logid:dbe40f87) Enabled non-edge zones available for scan: 1 2025-06-29 03:35:26,901 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3441852d]) (logid:dbe40f87) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:35:27,833 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6158596]) (logid:d4314e5b) No inactive management server node found 2025-06-29 03:35:27,833 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6158596]) (logid:d4314e5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:29,333 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0eaab15]) (logid:d0162005) No inactive management server node found 2025-06-29 03:35:29,333 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0eaab15]) (logid:d0162005) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:35:30,171 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) StorageCollector is running... 2025-06-29 03:35:30,176 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:35:30,177 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:35:30,177 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:35:30,177 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-6717118844223095510: Routed from 90520738888109 2025-06-29 03:35:30,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8234: Seq 7-8234: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:35:30,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8234 2025-06-29 03:35:30,232 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 4-6717118844223095510: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:35:30,235 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:35:30,236 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:35:30,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:35:30,236 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-3048092522799696817: Routed from 90520738888109 2025-06-29 03:35:30,283 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 1-3048092522799696817: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:35:30,293 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:35:30,293 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:35:30,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:35:30,294 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-1612007191621930099: Routed from 90520738888109 2025-06-29 03:35:30,305 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-1612007191621930099: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:35:30,318 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:35:30,318 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:35:30,318 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:35:30,319 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-3048092522799696818: Routed from 90520738888109 2025-06-29 03:35:30,493 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a01c0fce]) (logid:56e08ac7) HA health check task is running... 2025-06-29 03:35:30,821 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85fbccac]) (logid:fcadd88a) No inactive management server node found 2025-06-29 03:35:30,821 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85fbccac]) (logid:fcadd88a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:31,089 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:35:31,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:35:31,091 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) Seq 2-3048092522799696819: Routed from 90520738888109 2025-06-29 03:35:31,133 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) Seq 1-3048092522799696819: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:35:31,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:35:31,135 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) Seq 2-1612007191621930100: Routed from 90520738888109 2025-06-29 03:35:31,179 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-d5e08b4d]) (logid:7ef6554c) Seq 2-1612007191621930100: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:35:31,369 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 1-3048092522799696818: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:35:31,371 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:35:31,371 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:35:31,371 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:35:31,372 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-1612007191621930101: Routed from 90520738888109 2025-06-29 03:35:32,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c8e8f49]) (logid:da140753) No inactive management server node found 2025-06-29 03:35:32,334 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c8e8f49]) (logid:da140753) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:32,430 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-63604b85]) (logid:9d1d3a22) Seq 2-1612007191621930101: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:35:33,820 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b332c483]) (logid:efb3489a) No inactive management server node found 2025-06-29 03:35:33,820 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b332c483]) (logid:efb3489a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:33,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:35:33,984 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:35:33,985 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:35:33,987 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:35:34,495 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e8bbe13d]) (logid:a473e4b4) HA health check task is running... 2025-06-29 03:35:35,334 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b0ee63e]) (logid:74a305a9) No inactive management server node found 2025-06-29 03:35:35,335 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b0ee63e]) (logid:74a305a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:36,822 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94718b43]) (logid:e25b9722) No inactive management server node found 2025-06-29 03:35:36,822 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94718b43]) (logid:e25b9722) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:35:37,408 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,408 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SHA256SALT. 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NsxGuestNetworkGuru. 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImplicitDedicationPlanner. 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NsxPublicNetworkGuru. 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Nsx. 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,410 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RecreateHostAllocator. 2025-06-29 03:35:37,410 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirstFitAllocator. 2025-06-29 03:35:37,410 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Balance. 2025-06-29 03:35:37,410 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirstFitRouting. 2025-06-29 03:35:37,410 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TestingAllocator. 2025-06-29 03:35:37,410 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,411 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,411 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean MetricsServiceImpl. 2025-06-29 03:35:37,411 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,411 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,411 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,412 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,412 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean XCP Agent. 2025-06-29 03:35:37,413 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,413 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,413 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NonStrictHostAntiAffinityProcessor. 2025-06-29 03:35:37,413 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,415 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,415 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,415 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,415 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean staticpin. 2025-06-29 03:35:37,416 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecondaryStorageVmAlertAdapter. 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ovm3FenceBuilder. 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ovm3Guru. 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ovm3Investigator. 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BaremetalGuru. 2025-06-29 03:35:37,422 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NESTEDCLOUDSTACK. 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostAffinityProcessor. 2025-06-29 03:35:37,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,424 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,425 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetScalerControlCenterDaoImpl. 2025-06-29 03:35:37,425 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetScalerPodDaoImpl. 2025-06-29 03:35:37,425 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetScalerServicePackageDaoImpl. 2025-06-29 03:35:37,425 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetScalerVMManagerImpl. 2025-06-29 03:35:37,425 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Netscaler. 2025-06-29 03:35:37,427 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,427 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NonStrictHostAffinityProcessor. 2025-06-29 03:35:37,427 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,427 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,428 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,428 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,430 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,431 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean balanced. 2025-06-29 03:35:37,431 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,428 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,431 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,431 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,431 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CloudianConnectorImpl. 2025-06-29 03:35:37,431 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMware Discover. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AffinityGroupAccessChecker. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainChecker. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterAlertAdapter. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KVMHostActivityChecker. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KVMInvestigator. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KVMHAProvider. 2025-06-29 03:35:37,432 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,433 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,433 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApiRateLimitServiceImpl. 2025-06-29 03:35:37,433 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,436 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,436 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,436 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApiSolidFireIntegrationTestServiceImpl. 2025-06-29 03:35:37,436 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,436 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PaloAlto. 2025-06-29 03:35:37,437 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,441 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,441 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,441 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VersionDaoImpl. 2025-06-29 03:35:37,441 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,441 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SkipHeuresticsPlanner. 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HypervGuru. 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HypervInvestigator. 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,443 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BareMetalPlanner. 2025-06-29 03:35:37,443 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,443 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,443 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PrometheusExporterImpl. 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PrometheusExporterServerImpl. 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean totp. 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,444 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SspTenantDaoImpl. 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SspCredentialDaoImpl. 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SspUuidDaoImpl. 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StratosphereSsp. 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SspGuestNetworkGuru. 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean veeam. 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,446 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesSupportedVersionDaoImpl. 2025-06-29 03:35:37,447 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesClusterDetailsDaoImpl. 2025-06-29 03:35:37,447 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesClusterDaoImpl. 2025-06-29 03:35:37,447 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,447 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserConcentratedPodPlanner. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,447 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesClusterVmMapDaoImpl. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesVersionManagerImpl. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesClusterManagerImpl. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvsTunnelNetworkDaoImpl. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvsTunnelInterfaceDaoImpl. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KubernetesServiceHelper. 2025-06-29 03:35:37,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcDistributedRouterSeqNoDaoImpl. 2025-06-29 03:35:37,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvsNetworkTopologyGuruImpl. 2025-06-29 03:35:37,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvsTunnelManagerImpl. 2025-06-29 03:35:37,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvsGuestNetworkGuru. 2025-06-29 03:35:37,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ovs. 2025-06-29 03:35:37,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RandomAllocator. 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,452 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,453 INFO [c.c.h.h.m.HypervManagerImpl] (SpringContextShutdownHook:[]) (logid:) Cleanup mounted mount points used in current session 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ReconcileCommandDaoImpl. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RouterHealthCheckResultDaoImpl. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RoleDaoImpl. 2025-06-29 03:35:37,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeDataStoreDaoImpl. 2025-06-29 03:35:37,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserAccountDaoImpl. 2025-06-29 03:35:37,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMEntityDaoImpl. 2025-06-29 03:35:37,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CertificateDaoImpl. 2025-06-29 03:35:37,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageVPNUserDaoImpl. 2025-06-29 03:35:37,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AsyncJobJoinMapDaoImpl. 2025-06-29 03:35:37,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RemoteAccessVpnDetailsDaoImpl. 2025-06-29 03:35:37,455 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConfigurationSubGroupDaoImpl. 2025-06-29 03:35:37,455 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmTemplateDaoImpl. 2025-06-29 03:35:37,455 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PhysicalNetworkIsolationMethodDaoImpl. 2025-06-29 03:35:37,455 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenProviderDaoImpl. 2025-06-29 03:35:37,455 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EngineHostDaoImpl. 2025-06-29 03:35:37,456 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BgpPeerNetworkMapDaoImpl. 2025-06-29 03:35:37,456 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecondaryStorageHeuristicDaoImpl. 2025-06-29 03:35:37,456 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataCenterLinkLocalIpAddressDaoImpl. 2025-06-29 03:35:37,456 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LBHealthCheckPolicyDaoImpl. 2025-06-29 03:35:37,456 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CounterDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkOfferingJoinDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataCenterJoinDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupVMMapDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DiskOfferingDetailsDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TemplateDataStoreDaoImpl. 2025-06-29 03:35:37,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ObjectStoreDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirewallRuleDetailsDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StoragePoolWorkDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkServiceMapDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImageStoreJoinDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BgpPeerDetailsDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BaremetalPxeDaoImpl. 2025-06-29 03:35:37,458 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcOfferingServiceMapDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AccountGuestVlanMapDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenSecurityGroupRuleDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EngineHostPodDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GuestOSHypervisorDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkAsa1000vMapDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CustomHW Agent. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CiscoAsa1000vDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecondaryStorage. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CiscoVnmcDaoImpl. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CiscoVNMC. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SyncQueueDaoImpl. 2025-06-29 03:35:37,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Lxc Discover. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KVM Agent. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean dummyHostDiscoverer. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageSecurityGroupDaoImpl. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcOfferingDetailsDaoImpl. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GuestOsDetailsDaoImpl. 2025-06-29 03:35:37,460 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupJoinDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMRootDiskTagDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PodVlanDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AccountDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmCloneSettingDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkDetailsDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainJoinDaoImpl. 2025-06-29 03:35:37,461 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmJoinDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectInvitationJoinDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteVpnConnectionDetailsDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EngineDataCenterDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ServiceOfferingDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeReservationDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterDetailsDaoImpl. 2025-06-29 03:35:37,462 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkAccountDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DetailsDao. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DiskOfferingDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CommandExecLogDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteCustomerGatewayDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NicExtraDhcpOptionDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectRolePermissionsDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ServiceOfferingJoinDaoImpl. 2025-06-29 03:35:37,463 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserAccountJoinDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UcsManagerDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostGpuGroupsDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DcDetailsDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcOfferingDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostTransferMapDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ObjectInDataStoreDaoImpl. 2025-06-29 03:35:37,464 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConfigurationGroupDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcOfferingJoinDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StoragePoolAndAccessGroupMapDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GuestOSCategoryDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteVpnGatewayDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageNetworkIpRangeDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PortableIpRangeDaoImpl. 2025-06-29 03:35:37,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainVlanMapDaoImpl. 2025-06-29 03:35:37,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScalePolicyConditionMapDaoImpl. 2025-06-29 03:35:37,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageDaoImpl. 2025-06-29 03:35:37,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean InstanceGroupVMMapDaoImpl. 2025-06-29 03:35:37,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerStatusDaoImpl. 2025-06-29 03:35:37,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotPolicyDetailsDaoImpl. 2025-06-29 03:35:37,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ipv4GuestSubnetNetworkMapDaoImpl. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterDrsPlanDaoImpl. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PrimaryDataStoreDetailsDaoImpl. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KeystoreDaoImpl. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RouterNetworkDaoImpl. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LXCGuru. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KVMGuru. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageEventDetailsDaoImpl. 2025-06-29 03:35:37,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageEventDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteVpnConnectionDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DirectDownloadCertificateDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StoragePoolHostDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotJoinDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RemoteAccessVpnDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostDetailsDaoImpl. 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,468 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkExternalLoadBalancerDaoImpl. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConsoleProxyDaoImpl. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PLAINTEXT. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeJoinDaoImpl. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmwareDatacenterZoneMapDaoImpl. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DynamicRoleBasedAPIAccessChecker. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageNetworkIpAddressDaoImpl. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ip Alloc. 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMInstanceDaoImpl. 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EventDaoImpl. 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StoragePoolJoinDaoImpl. 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ASNumberRangeDaoImpl. 2025-06-29 03:35:37,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UcsBladeDaoImpl. 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectRoleBasedApiAccessChecker. 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMReservationDaoImpl. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainDaoImpl. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcGatewayDaoImpl. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ExplicitDedicationProcessor. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PassphraseDaoImpl. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ScaleIOVMSnapshotStrategy. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DefaultVMSnapshotStrategy. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageVMSnapshotStrategy. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean InstanceGroupDaoImpl. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectInvitationDaoImpl. 2025-06-29 03:35:37,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VlanDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceLimitDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean InstanceGroupJoinDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LoadBalancerCertMapDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KVMFenceBuilder. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RecreatableFencer. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotDetailsDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LdapConfigurationDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageStorageDaoImpl. 2025-06-29 03:35:37,472 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean REDFISH. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserIpAddressDetailsDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LdapTrustMapDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ipv6GuestPrefixSubnetNetworkMapDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LDAP. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CapacityDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LdapManagerImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LaunchPermissionDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BaremetalRctDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcServiceMapDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageNetworkDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GlobalLoadBalancerDaoImpl. 2025-06-29 03:35:37,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LBHealthCheckPolicyDetailsDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BucketDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean XenServerFenceBuilder. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean XenServerGuru. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DiskOfferingJoinDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BrocadeVcsDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NicSecondaryIpDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BrocadeVcsNetworkVlanMappingDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BrocadeVcsGuestNetworkGuru. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BrocadeVcs. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkOfferingDaoImpl. 2025-06-29 03:35:37,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImageStoreObjectDownloadDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataCenterGuestIpv6PrefixDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerHostPeerJoinDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectRoleDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean IPAddressDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DedicatedResourceDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SSHKeyPairDaoImpl. 2025-06-29 03:35:37,475 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmDeployAsIsDetailsDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserIpv6AddressDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean networker. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageVMInstanceDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PBKDF2. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BgpPeerDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AffinityGroupDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMScheduleDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HAConfigDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenFabricLBHealthMonitorDaoImpl. 2025-06-29 03:35:37,476 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmProfileDaoImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenProviderServiceImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenServiceImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenGuestNetworkGuru. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenElement. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostTagsDaoImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NsxProviderDaoImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmDaoImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ServiceOfferingDetailsDaoImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkPermissionDaoImpl. 2025-06-29 03:35:37,477 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LBStickinessPolicyDetailsDaoImpl. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DirectDownloadCertificateHostMapDaoImpl. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkDaoImpl. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostAntiAffinityProcessor. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OutOfBandManagementDaoImpl. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StaticRoleBasedAPIAccessChecker. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLItemDetailsDaoImpl. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserDaoImpl. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean managementServerMaintenanceManager. 2025-06-29 03:35:37,478 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AccountDetailsDaoImpl. 2025-06-29 03:35:37,479 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PortForwardingRulesDaoImpl. 2025-06-29 03:35:37,479 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AsyncJobDaoImpl. 2025-06-29 03:35:37,479 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeStatsDaoImpl. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ContrailManagerImpl. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ContrailVpcElement. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ContrailElement. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean elastistorVolumeApiService. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupRuleDaoImpl. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApplicationLoadBalancerRuleDaoImpl. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ExternalFirewallDeviceDaoImpl. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ContrailGuru. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GloboDnsElement. 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,480 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VirtualRouterProviderDaoImpl. 2025-06-29 03:35:37,481 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PlannerHostReservationDaoImpl. 2025-06-29 03:35:37,481 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostDaoImpl. 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BigSwitchBcfDaoImpl. 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PodVlanMapDaoImpl. 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BigSwitchBcf. 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BigSwitchBcfGuestNetworkGuru. 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UploadDaoImpl. 2025-06-29 03:35:37,482 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserDetailsDaoImpl. 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerHostPeerDaoImpl. 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostTagsDaoImpl. 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupWorkDaoImpl. 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmProfileDetailsDaoImpl. 2025-06-29 03:35:37,483 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BucketStatisticsDaoImpl. 2025-06-29 03:35:37,484 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NicIpAliasDaoImpl. 2025-06-29 03:35:37,484 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SharedFSDaoImpl. 2025-06-29 03:35:37,484 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMTemplateDaoImpl. 2025-06-29 03:35:37,484 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean WebhookDeliveryDaoImpl. 2025-06-29 03:35:37,484 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean WebhookJoinDaoImpl. 2025-06-29 03:35:37,484 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean WebhookDeliveryJoinDaoImpl. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageNetworkOfferingDaoImpl. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean WebhookDaoImpl. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean webhookEventBus. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean WebhookServiceImpl. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean WebhookApiServiceImpl. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Hyper-V Agent. 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,485 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AffinityGroupVMMapDaoImpl. 2025-06-29 03:35:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConditionDaoImpl. 2025-06-29 03:35:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VxlanGuestNetworkGuru. 2025-06-29 03:35:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConsoleSessionDaoImpl. 2025-06-29 03:35:37,487 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,487 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectAccountJoinDaoImpl. 2025-06-29 03:35:37,487 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OauthProviderDaoImpl. 2025-06-29 03:35:37,487 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OAUTH2Auth. 2025-06-29 03:35:37,487 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean oauth2. 2025-06-29 03:35:37,488 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean github. 2025-06-29 03:35:37,488 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean google. 2025-06-29 03:35:37,488 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorPoolStatsCollector. 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorPoolAbandonObjectsCollector. 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorPoolVMSnapshotStrategy. 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ElasticLbVmMapDaoImpl. 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ElasticLoadBalancerManagerImpl. 2025-06-29 03:35:37,489 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,490 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmGroupVmMapDaoImpl. 2025-06-29 03:35:37,492 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LegacyZoneDaoImpl. 2025-06-29 03:35:37,492 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,493 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OpRouterMonitorServiceDaoImpl. 2025-06-29 03:35:37,494 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkRuleConfigDaoImpl. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmWorkJobDaoImpl. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataCenterDaoImpl. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OpenDaylightControllerMappingDaoImpl. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OpendaylightGuestNetworkGuru. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OpendaylightElement. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainRouterDaoImpl. 2025-06-29 03:35:37,495 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmDiskStatisticsDaoImpl. 2025-06-29 03:35:37,494 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NiciraNvpDaoImpl. 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NiciraNvpNicMappingDaoImpl. 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NiciraNvpRouterMappingDaoImpl. 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageIPAddressDaoImpl. 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NiciraNvpGuestNetworkGuru. 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NiciraNvp. 2025-06-29 03:35:37,496 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,497 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,497 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,498 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GlobalLoadBalancerLbRuleMapDaoImpl. 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean dummy. 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VnfTemplateNicDaoImpl. 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HypervisorAdapter. 2025-06-29 03:35:37,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,500 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,500 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SHAREDFSVM. 2025-06-29 03:35:37,500 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,502 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AffinityGroupJoinDaoImpl. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMwareFenceBuilder. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMwareGuru. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMwareInvestigator. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BareMetalPxe. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BareMetalUserdata. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BareMetalDhcp. 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,503 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,504 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkDomainDaoImpl. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecondaryStorageVmDaoImpl. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PhysicalNetworkTagDaoImpl. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PublicNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DirectPodBasedNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VirtualRouter. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PodBasedNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ControlNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConfigDrive. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PrivateNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupProvider. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ExternalGuestNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DirectNetworkGuru. 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,505 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMScheduledJobDaoImpl. 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageLoadBalancerPolicyDaoImpl. 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerHostDaoImpl. 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean MD5. 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CiscoNexus1000vVSM. 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostJoinDaoImpl. 2025-06-29 03:35:37,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AnnotationDaoImpl. 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImageStoreDetailsDaoImpl. 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LoadBalancerVMMapDaoImpl. 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLDaoImpl. 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RootCAProvider. 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaEmailTemplatesDaoImpl. 2025-06-29 03:35:37,507 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmGroupPolicyMapDaoImpl. 2025-06-29 03:35:37,508 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaTariffDaoImpl. 2025-06-29 03:35:37,508 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SharedFSJoinDaoImpl. 2025-06-29 03:35:37,508 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaCreditsDaoImpl. 2025-06-29 03:35:37,508 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmDetailsDaoImpl. 2025-06-29 03:35:37,508 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerJoinDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaEmailConfigurationDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaAccountDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageVpcDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaBalanceDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmDetailsDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaUsageDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaStatementImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.q.QuotaStatementImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Statement Manager 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaManagerImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.q.QuotaManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Quota Manager 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaAlertManagerImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.q.QuotaAlertManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Alert Manager 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QuotaServiceImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PrivateIpDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.q.QuotaStatementImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Statement Manager 2025-06-29 03:35:37,509 INFO [o.a.c.q.QuotaManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Quota Manager 2025-06-29 03:35:37,509 INFO [o.a.c.q.QuotaAlertManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Alert Manager 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataCenterDetailsDaoImpl. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean nas. 2025-06-29 03:35:37,509 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VlanDetailsDaoImpl. 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScalePolicyDaoImpl. 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LoadBalancerDaoImpl. 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserStatsLogDaoImpl. 2025-06-29 03:35:37,510 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcDetailsDaoImpl. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmwareDatacenterDaoImpl. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SAMLTokenDaoImpl. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirewallRulesCidrsDaoImpl. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SAML2Auth. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SAML2. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMSnapshotDaoImpl. 2025-06-29 03:35:37,511 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotZoneDaoImpl. 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectJoinDaoImpl. 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PortProfileDaoImpl. 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PublicIpQuarantineDaoImpl. 2025-06-29 03:35:37,512 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PhysicalNetworkServiceProviderDaoImpl. 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotDataStoreDaoImpl. 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvmGuru. 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvmFenceBuilder. 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMSnapshotDetailsDaoImpl. 2025-06-29 03:35:37,513 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,514 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RolePermissionsDaoImpl. 2025-06-29 03:35:37,514 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostPodDaoImpl. 2025-06-29 03:35:37,515 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ItWorkDaoImpl. 2025-06-29 03:35:37,516 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean vnet Alloc. 2025-06-29 03:35:37,516 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,516 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConsoleProxyAlertAdapter. 2025-06-29 03:35:37,516 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,516 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HostDetailsDaoImpl. 2025-06-29 03:35:37,517 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImageStoreDaoImpl. 2025-06-29 03:35:37,517 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VGPUTypesDaoImpl. 2025-06-29 03:35:37,517 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserDataDaoImpl. 2025-06-29 03:35:37,518 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpnUserDaoImpl. 2025-06-29 03:35:37,518 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmStatsDaoImpl. 2025-06-29 03:35:37,518 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageVolumeDaoImpl. 2025-06-29 03:35:37,518 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackupDaoImpl. 2025-06-29 03:35:37,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceIconDaoImpl. 2025-06-29 03:35:37,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcDaoImpl. 2025-06-29 03:35:37,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HighAvailabilityDaoImpl. 2025-06-29 03:35:37,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StoragePoolTagsDaoImpl. 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLItemDaoImpl. 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean condensed. 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeDetailsDaoImpl. 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EngineClusterDaoImpl. 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteCustomerGatewayDetailsDaoImpl. 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,520 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ovm Discover. 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AsyncJobJoinDaoImpl. 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CrlDaoImpl. 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataCenterIpv4GuestSubnetDaoImpl. 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ObjectStoreDetailsDaoImpl. 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PrimaryDataStoreDaoImpl. 2025-06-29 03:35:37,521 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecurityGroupRulesDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PhysicalNetworkTrafficTypeDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceTagJoinDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkOpDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLListDetailsDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ReservationDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackupScheduleDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageVmDiskDaoImpl. 2025-06-29 03:35:37,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLItemCidrsDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmGroupDetailsDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMComputeTagDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NicDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserStatisticsDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AlertDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterVSMMapDaoImpl. 2025-06-29 03:35:37,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SyncQueueItemDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TemplateDeployAsIsDetailsDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceTagsDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectAccountDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SslCertDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TemplateJoinDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ASNumberDaoImpl. 2025-06-29 03:35:37,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcGatewayDetailsDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VnfTemplateDetailsDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotPolicyDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkExternalFirewallDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMTemplatePoolDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SecondaryStorageVmDefaultAllocator. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotScheduleDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ovm3Discover. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageNetworksDaoImpl. 2025-06-29 03:35:37,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,526 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageBackupDaoImpl. 2025-06-29 03:35:37,526 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StaticRouteDaoImpl. 2025-06-29 03:35:37,526 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AccountVlanMapDaoImpl. 2025-06-29 03:35:37,526 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackupOfferingDaoImpl. 2025-06-29 03:35:37,526 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageJobDaoImpl. 2025-06-29 03:35:37,526 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GuestOSDaoImpl. 2025-06-29 03:35:37,527 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AsyncJobJournalDaoImpl. 2025-06-29 03:35:37,527 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainDetailsDaoImpl. 2025-06-29 03:35:37,527 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EventJoinDaoImpl. 2025-06-29 03:35:37,527 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean MonitoringServiceDaoImpl. 2025-06-29 03:35:37,527 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AffinityGroupDomainMapDaoImpl. 2025-06-29 03:35:37,528 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirewallRulesDaoImpl. 2025-06-29 03:35:37,528 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectDaoImpl. 2025-06-29 03:35:37,528 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NicDetailsDaoImpl. 2025-06-29 03:35:37,528 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LBStickinessPolicyDaoImpl. 2025-06-29 03:35:37,529 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OvsProviderDaoImpl. 2025-06-29 03:35:37,529 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkOfferingServiceMapDaoImpl. 2025-06-29 03:35:37,529 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PhysicalNetworkDaoImpl. 2025-06-29 03:35:37,529 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TungstenGuestNetworkIpAddressDaoImpl. 2025-06-29 03:35:37,530 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeDaoImpl. 2025-06-29 03:35:37,530 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmGroupDaoImpl. 2025-06-29 03:35:37,530 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleVmGroupStatisticsDaoImpl. 2025-06-29 03:35:37,530 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainRouterJoinDaoImpl. 2025-06-29 03:35:37,530 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AccountJoinDaoImpl. 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterDrsPlanMigrationDaoImpl. 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMTemplateZoneDaoImpl. 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserDispersingPlanner. 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BareMetalAdapter. 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceCountDaoImpl. 2025-06-29 03:35:37,531 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VsphereStoragePolicyDaoImpl. 2025-06-29 03:35:37,532 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkOfferingDetailsDaoImpl. 2025-06-29 03:35:37,532 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackupRepositoryDaoImpl. 2025-06-29 03:35:37,532 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean InlineLoadBalancerNicMapDaoImpl. 2025-06-29 03:35:37,532 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ExternalLoadBalancerDeviceDaoImpl. 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteVpnGatewayDetailsDaoImpl. 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMTemplateDetailsDaoImpl. 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PortableIpDaoImpl. 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterDaoImpl. 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BaremetalDhcpDaoImpl. 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,533 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RegionDaoImpl. 2025-06-29 03:35:37,534 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CiscoNexusVSMDeviceDaoImpl. 2025-06-29 03:35:37,534 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmRulesetLogDaoImpl. 2025-06-29 03:35:37,534 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirewallRulesDcidrsDaoImpl. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LinstorVMSnapshotStrategy. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMNetworkMapDaoImpl. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsagePortForwardingRuleDaoImpl. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HypervisorCapabilitiesDaoImpl. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerNode. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConfigurationServerImpl. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DatabaseIntegrityChecker. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AsyncJobManagerImpl. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirstFitPlanner. 2025-06-29 03:35:37,535 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectManagerImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkOrchestrator. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageOrchestrator. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotSchedulerImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AutoScaleManagerImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AccountManagerImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackupRepositoryServiceImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkServiceImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UsageServiceImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApiServer. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BareMetalPingServiceImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean FirewallManagerImpl. 2025-06-29 03:35:37,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmwareManagerImpl. 2025-06-29 03:35:37,537 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:[]) (logid:) shutting down scheduled tasks 2025-06-29 03:35:37,537 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:[]) (logid:) Cleanup mounted NFS mount points used in current session 2025-06-29 03:35:37,537 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean IpAddressManagerImpl. 2025-06-29 03:35:37,537 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusteredAgentManagerImpl. 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LocalStorage. 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserFirst. 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean GCStorage. 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterScopeStoragePoolAllocator. 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ZoneWideStoragePoolAllocator. 2025-06-29 03:35:37,539 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,541 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,541 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean IPMITOOL. 2025-06-29 03:35:37,541 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,542 INFO [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) Disconnecting agents: 4 2025-06-29 03:35:37,542 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,543 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,543 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:37,543 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,547 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (JettyShutdownThread:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,547 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (JettyShutdownThread:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,548 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PingInvestigator. 2025-06-29 03:35:37,548 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SimpleInvestigator. 2025-06-29 03:35:37,548 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean XenServerInvestigator. 2025-06-29 03:35:37,548 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementIPSysVMInvestigator. 2025-06-29 03:35:37,548 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,556 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,556 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Bare Metal Agent. 2025-06-29 03:35:37,556 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,557 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Stopping CloudStack Components 2025-06-29 03:35:37,557 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApiSiocServiceImpl. 2025-06-29 03:35:37,557 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:37,566 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:37,586 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:37,605 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:37,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean QueryManagerImpl. 2025-06-29 03:35:37,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HighAvailabilityManagerExtImpl. 2025-06-29 03:35:37,616 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (SpringContextShutdownHook:[]) (logid:) Wakeup workers HA 2025-06-29 03:35:37,617 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-1:[]) (logid:) Time to go home! 2025-06-29 03:35:37,617 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-2:[]) (logid:) Time to go home! 2025-06-29 03:35:37,617 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-3:[]) (logid:) Time to go home! 2025-06-29 03:35:37,617 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[]) (logid:) Time to go home! 2025-06-29 03:35:37,617 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[]) (logid:) Time to go home! 2025-06-29 03:35:37,624 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VolumeApiServiceImpl. 2025-06-29 03:35:37,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VnfTemplateManagerImpl. 2025-06-29 03:35:37,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HypervisorGuruManagerImpl. 2025-06-29 03:35:37,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ReconcileCommandServiceImpl. 2025-06-29 03:35:37,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackgroundPollManagerImpl. 2025-06-29 03:35:37,626 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImageStoreServiceImpl. 2025-06-29 03:35:37,626 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DataStoreProviderManagerImpl. 2025-06-29 03:35:37,626 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UploadMonitorImpl. 2025-06-29 03:35:37,626 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BackupManagerImpl. 2025-06-29 03:35:37,627 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ImageStoreUploadMonitorImpl. 2025-06-29 03:35:37,627 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RemoteAccessVpnManagerImpl. 2025-06-29 03:35:37,627 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Site2SiteVpnManagerImpl. 2025-06-29 03:35:37,628 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserVmManagerImpl. 2025-06-29 03:35:37,628 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMLeaseManagerImpl. 2025-06-29 03:35:37,629 DEBUG [o.a.c.v.l.VMLeaseManagerImpl] (SpringContextShutdownHook:[]) (logid:) Removing Lease subscriber for async job events 2025-06-29 03:35:37,629 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DomainManagerImpl. 2025-06-29 03:35:37,629 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLManagerImpl. 2025-06-29 03:35:37,629 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean PremiumSecondaryStorageManagerImpl. 2025-06-29 03:35:37,631 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMSnapshotManagerImpl. 2025-06-29 03:35:37,631 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Volume Manager. 2025-06-29 03:35:37,631 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConfigurationManagerImpl. 2025-06-29 03:35:37,631 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConsoleAccessManagerImpl. 2025-06-29 03:35:37,632 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DownloadMonitorImpl. 2025-06-29 03:35:37,632 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceMetaDataManagerImpl. 2025-06-29 03:35:37,632 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserPasswordResetManagerImpl. 2025-06-29 03:35:37,632 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterFenceManagerImpl. 2025-06-29 03:35:37,632 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,632 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,633 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VirtualNetworkApplianceManagerImpl. 2025-06-29 03:35:37,633 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DiagnosticsServiceImpl. 2025-06-29 03:35:37,633 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,633 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (SpringContextShutdownHook:[]) (logid:) HealthCheckmanager is getting Stopped 2025-06-29 03:35:37,633 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean HAManagerImpl. 2025-06-29 03:35:37,638 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcManagerImpl. 2025-06-29 03:35:37,638 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Baremetal Vlan Manager. 2025-06-29 03:35:37,638 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BaremetalPxeManagerImpl. 2025-06-29 03:35:37,638 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SharedFSServiceImpl. 2025-06-29 03:35:37,639 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AsyncJobMonitor. 2025-06-29 03:35:37,639 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMScheduleManagerImpl. 2025-06-29 03:35:37,639 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ipv6AddressManagerImpl. 2025-06-29 03:35:37,639 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkUsageManagerImpl. 2025-06-29 03:35:37,639 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ConsoleProxyManagerImpl. 2025-06-29 03:35:37,639 INFO [c.c.c.ConsoleProxyManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stop console proxy manager 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AlertManagerImpl. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageBrowserImpl. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean APIAuthenticationManagerImpl. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VMSchedulerImpl. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Baremetal Manager. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CapacityManagerImpl. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ExternalNetworkDeviceManagerImpl. 2025-06-29 03:35:37,640 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BucketApiServiceImpl. 2025-06-29 03:35:37,641 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusteredVirtualMachineManagerImpl. 2025-06-29 03:35:37,641 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TaggedResourceManagerImpl. 2025-06-29 03:35:37,641 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,642 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceLimitManagerImpl. 2025-06-29 03:35:37,643 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SnapshotManagerImpl. 2025-06-29 03:35:37,643 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkModelImpl. 2025-06-29 03:35:37,643 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean AnnotationManagerImpl. 2025-06-29 03:35:37,643 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CAManagerImpl. 2025-06-29 03:35:37,644 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean KeystoreManagerImpl. 2025-06-29 03:35:37,644 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean managementServerMaintenanceManager. 2025-06-29 03:35:37,646 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StatsCollector. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean UserDataManagerImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean InternalLoadBalancerVMManagerImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RulesManagerImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceManagerImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RollingMaintenanceManagerImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean TemplateManagerImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BaremetalKickStartServiceImpl. 2025-06-29 03:35:37,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,655 DEBUG [o.a.c.o.d.OutOfBandManagementDaoImpl] (SpringContextShutdownHook:[]) (logid:) Expired out-of-band management ownership for hosts owned by management server id:90520738888109 2025-06-29 03:35:37,655 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean com.cloud.network.security.SecurityGroupManagerImpl2. 2025-06-29 03:35:37,655 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceCleanupServiceImpl. 2025-06-29 03:35:37,655 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DeploymentPlanningManagerImpl. 2025-06-29 03:35:37,655 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterDrsServiceImpl. 2025-06-29 03:35:37,655 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean null. 2025-06-29 03:35:37,655 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean OCFS2ManagerImpl. 2025-06-29 03:35:37,656 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RoleManagerImpl. 2025-06-29 03:35:37,656 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DirectDownloadManagerImpl. 2025-06-29 03:35:37,656 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApplicationLoadBalancerManagerImpl. 2025-06-29 03:35:37,656 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean LoadBalancingRulesManagerImpl. 2025-06-29 03:35:37,656 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean EventDistributorImpl. 2025-06-29 03:35:37,656 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean NetworkACLServiceImpl. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcVirtualNetworkApplianceManagerImpl. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ProjectRoleManagerImpl. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean StorageNetworkManagerImpl. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ResourceIconManagerImpl. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean SyncQueueManagerImpl. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean BASIC. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmWorkJobWakeupDispatcher. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VmWorkJobDispatcher. 2025-06-29 03:35:37,657 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterManagerImpl. 2025-06-29 03:35:37,657 INFO [c.c.c.ClusterManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Cluster manager, msid : 90520738888109, runId : 1751116087255, host : ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:35:38,701 INFO [c.c.c.ClusterManagerImpl] (SpringContextShutdownHook:[]) (logid:) Cluster manager is stopped 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Basic. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean DumpRegistry. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApiAsyncJobDispatcher. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ApiDiscoveryServiceImpl. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean RoutedIpv4ManagerImpl. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean VpcVirtualRouter. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean InternalLbVm. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean CloudZonesStartupProcessor. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterServiceServletAdapter. 2025-06-29 03:35:38,701 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean IndirectAgentLBServiceImpl. 2025-06-29 03:35:38,702 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean Ipv6ServiceImpl. 2025-06-29 03:35:38,702 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean cloud-init. 2025-06-29 03:35:38,702 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterBasedAgentLoadBalancerPlanner. 2025-06-29 03:35:38,702 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ManagementServerImpl. 2025-06-29 03:35:38,702 ERROR [c.c.c.C.ListenerThread] (Thread-24:[]) (logid:) Unexpected exception java.net.SocketException: Socket closed at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:694) at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:767) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:675) at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:641) at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:596) at java.base/sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:271) at com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.run(ClusterServiceServletContainer.java:180) 2025-06-29 03:35:38,702 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) Done Stopping CloudStack Components 2025-06-29 03:35:38,703 INFO [c.c.h.h.m.HypervManagerImpl] (SpringContextShutdownHook:[]) (logid:) Cleanup mounted mount points used in current session 2025-06-29 03:35:38,708 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:[]) (logid:) shutting down scheduled tasks 2025-06-29 03:35:38,708 INFO [c.c.h.v.m.VmwareManagerImpl] (SpringContextShutdownHook:[]) (logid:) Cleanup mounted NFS mount points used in current session 2025-06-29 03:35:38,710 INFO [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) Disconnecting agents: 4 2025-06-29 03:35:38,712 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:38,737 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:38,748 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:38,759 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (SpringContextShutdownHook:[]) (logid:) [Resource state = Enabled, Agent event = , Host = ManagementServerDown] 2025-06-29 03:35:38,771 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (SpringContextShutdownHook:[]) (logid:) Wakeup workers HA 2025-06-29 03:35:38,773 DEBUG [o.a.c.v.l.VMLeaseManagerImpl] (SpringContextShutdownHook:[]) (logid:) Removing Lease subscriber for async job events 2025-06-29 03:35:38,775 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:[]) (logid:) Unmatched Global lock secstorage.scan.lock reference usage detected, check your code! 2025-06-29 03:35:38,775 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:[]) (logid:) Unmatched Global lock secStorageVm.alloc reference usage detected, check your code! 2025-06-29 03:35:38,775 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (SpringContextShutdownHook:[]) (logid:) HealthCheckmanager is getting Stopped 2025-06-29 03:35:38,777 INFO [c.c.c.ConsoleProxyManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stop console proxy manager 2025-06-29 03:35:38,777 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:[]) (logid:) Unmatched Global lock consoleproxy.scan.lock reference usage detected, check your code! 2025-06-29 03:35:38,777 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:[]) (logid:) Unmatched Global lock consoleproxy.alloc reference usage detected, check your code! 2025-06-29 03:35:38,781 DEBUG [o.a.c.o.d.OutOfBandManagementDaoImpl] (SpringContextShutdownHook:[]) (logid:) Expired out-of-band management ownership for hosts owned by management server id:90520738888109 2025-06-29 03:35:38,781 INFO [c.c.c.ClusterManagerImpl] (SpringContextShutdownHook:[]) (logid:) Stopping Cluster manager, msid : 90520738888109, runId : 1751116087255, host : ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:35:39,702 INFO [c.c.c.C.ListenerThread] (Thread-24:[]) (logid:) Cluster service servlet container shutdown 2025-06-29 03:35:39,805 INFO [c.c.c.ClusterManagerImpl] (SpringContextShutdownHook:[]) (logid:) Cluster manager is stopped 2025-06-29 03:35:39,806 INFO [o.a.c.s.NfsMountManagerImpl] (SpringContextShutdownHook:[]) (logid:) Clean up mounted NFS mount points used in current session. 2025-06-29 03:35:39,808 DEBUG [o.a.c.s.NfsMountManagerImpl] (SpringContextShutdownHook:[]) (logid:) Executing command [sudo mount ]. 2025-06-29 03:35:40,152 DEBUG [o.a.c.s.NfsMountManagerImpl] (SpringContextShutdownHook:[]) (logid:) Successfully executed process [4064495] for command [sudo mount ]. 2025-06-29 03:35:40,156 DEBUG [o.a.c.s.NfsMountManagerImpl] (SpringContextShutdownHook:[]) (logid:) Executing command [sudo umount /var/cloudstack/mnt/90520738888109.7d16e2bc ]. 2025-06-29 03:35:40,221 DEBUG [o.a.c.s.NfsMountManagerImpl] (SpringContextShutdownHook:[]) (logid:) Successfully executed process [4064510] for command [sudo umount /var/cloudstack/mnt/90520738888109.7d16e2bc ]. 2025-06-29 03:35:40,221 DEBUG [o.a.c.s.NfsMountManagerImpl] (SpringContextShutdownHook:[]) (logid:) Successfully umounted /var/cloudstack/mnt/90520738888109.7d16e2bc 2025-06-29 03:35:42,074 INFO [o.a.c.ServerDaemon] (main:[]) (logid:) Server configuration file found: /etc/cloudstack/management/server.properties 2025-06-29 03:35:42,083 DEBUG [c.c.u.c.EncryptionSecretKeyChecker] (main:[]) (logid:) Encryption Type: null 2025-06-29 03:35:42,090 INFO [o.a.c.ServerDaemon] (main:[]) (logid:) Initializing server daemon on null, with http.enable=true, http.port=8080, https.enable=false, https.port=8443, context.path=/client 2025-06-29 03:35:53,672 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bootstrap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context.xml] 2025-06-29 03:35:53,672 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bootstrap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:35:53,826 DEBUG [c.c.u.c.EncryptionSecretKeyChecker] (main:[]) (logid:) Encryption Type: file 2025-06-29 03:35:53,829 DEBUG [c.c.u.c.CloudStackEncryptor] (main:[]) (logid:) Calling to initialize for class com.cloud.utils.crypt.EncryptionSecretKeyChecker 2025-06-29 03:35:54,066 DEBUG [c.c.u.c.CloudStackEncryptor] (main:[]) (logid:) Initialized with all possible encryptors 2025-06-29 03:35:54,086 DEBUG [c.c.u.c.CloudStackEncryptor] (main:[]) (logid:) CloudStack will encrypt and decrypt values using encryptor : AeadBase64Encryptor for class EncryptionSecretKeyChecker 2025-06-29 03:35:54,192 INFO [c.c.u.LogUtils] (main:[]) (logid:) log4j configuration found at /etc/cloudstack/management/log4j-cloud.xml 2025-06-29 03:35:54,198 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [bootstrap] in 521 ms 2025-06-29 03:35:54,199 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: bootstrap 2025-06-29 03:35:54,199 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: system 2025-06-29 03:35:54,199 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: core 2025-06-29 03:35:54,199 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: allocator 2025-06-29 03:35:54,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: host-allocator-random 2025-06-29 03:35:54,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: planner 2025-06-29 03:35:54,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: api-planner 2025-06-29 03:35:54,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: baremetal-planner 2025-06-29 03:35:54,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: explicit-dedication 2025-06-29 03:35:54,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: host-affinity 2025-06-29 03:35:54,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: host-anti-affinity 2025-06-29 03:35:54,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: implicit-dedication 2025-06-29 03:35:54,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: non-strict-host-affinity 2025-06-29 03:35:54,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: non-strict-host-anti-affinity 2025-06-29 03:35:54,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-planner 2025-06-29 03:35:54,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: skip-heurestics 2025-06-29 03:35:54,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: user-concentrated-pod 2025-06-29 03:35:54,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-allocator 2025-06-29 03:35:54,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: api 2025-06-29 03:35:54,203 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: acl-dynamic-role-based 2025-06-29 03:35:54,203 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: acl-project-role-based 2025-06-29 03:35:54,203 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: acl-static-role-based 2025-06-29 03:35:54,203 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: cloudian 2025-06-29 03:35:54,203 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ldap 2025-06-29 03:35:54,203 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: maintenance 2025-06-29 03:35:54,204 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: md5 2025-06-29 03:35:54,204 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: metrics 2025-06-29 03:35:54,204 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: oauth2 2025-06-29 03:35:54,204 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: pbkdf2 2025-06-29 03:35:54,204 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: plaintext 2025-06-29 03:35:54,204 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: prometheus 2025-06-29 03:35:54,205 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: quota 2025-06-29 03:35:54,205 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: rate-limit 2025-06-29 03:35:54,205 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: saml2 2025-06-29 03:35:54,205 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-api 2025-06-29 03:35:54,205 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: sha256salted 2025-06-29 03:35:54,205 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: solidfire-intg-test 2025-06-29 03:35:54,206 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: staticpin 2025-06-29 03:35:54,206 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: totp 2025-06-29 03:35:54,206 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vmware-sioc 2025-06-29 03:35:54,206 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: api-config 2025-06-29 03:35:54,206 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: backend 2025-06-29 03:35:54,206 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: backup 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: dummy-backup 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: nas 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: networker 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: veeam 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ca 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: root-ca 2025-06-29 03:35:54,207 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: compute 2025-06-29 03:35:54,208 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: baremetal-compute 2025-06-29 03:35:54,208 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: hyperv-compute 2025-06-29 03:35:54,208 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: kubernetes 2025-06-29 03:35:54,208 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: kubernetes-service 2025-06-29 03:35:54,208 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: kvm-compute 2025-06-29 03:35:54,208 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ovm-compute 2025-06-29 03:35:54,209 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ovm3-compute 2025-06-29 03:35:54,209 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-alert-adapter-compute 2025-06-29 03:35:54,209 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-compute 2025-06-29 03:35:54,209 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-fencer 2025-06-29 03:35:54,209 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-investigator 2025-06-29 03:35:54,209 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vmware-compute 2025-06-29 03:35:54,210 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: xenserver-compute 2025-06-29 03:35:54,210 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: network 2025-06-29 03:35:54,210 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: baremetal-network 2025-06-29 03:35:54,210 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: bigswitch 2025-06-29 03:35:54,210 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: cisco-vnmc 2025-06-29 03:35:54,210 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: contrail 2025-06-29 03:35:54,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: elb 2025-06-29 03:35:54,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: globodns 2025-06-29 03:35:54,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: netscaler 2025-06-29 03:35:54,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: nsx 2025-06-29 03:35:54,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: nvp 2025-06-29 03:35:54,212 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: opendaylight 2025-06-29 03:35:54,212 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ovs 2025-06-29 03:35:54,212 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: paloalto 2025-06-29 03:35:54,212 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-network 2025-06-29 03:35:54,212 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ssp 2025-06-29 03:35:54,212 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: tungsten 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vcs 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vmware-network 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vxlan 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-alert-adapter-backend 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: baremetal-storage 2025-06-29 03:35:54,213 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-alert-adapter-storage 2025-06-29 03:35:54,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-storage 2025-06-29 03:35:54,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-template-adapter 2025-06-29 03:35:54,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: sharedfs-provider-storagevm 2025-06-29 03:35:54,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-allocator 2025-06-29 03:35:54,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-image-default 2025-06-29 03:35:54,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-image-s3 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-image-swift 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-object-ceph 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-object-cloudian 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-object-minio 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-object-simulator 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-cloudbyte 2025-06-29 03:35:54,215 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-datera 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-default 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-flasharray 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-linstor 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-primera 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-scaleio 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-solidfire 2025-06-29 03:35:54,216 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: storage-volume-storpool 2025-06-29 03:35:54,217 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vmware-storage 2025-06-29 03:35:54,217 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: cluster 2025-06-29 03:35:54,217 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: balanced 2025-06-29 03:35:54,217 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: condensed 2025-06-29 03:35:54,217 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: discoverer 2025-06-29 03:35:54,217 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: baremetal-discoverer 2025-06-29 03:35:54,218 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: hyperv-discoverer 2025-06-29 03:35:54,218 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ovm-discoverer 2025-06-29 03:35:54,218 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ovm3-discoverer 2025-06-29 03:35:54,218 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: secondary-storage-discoverer 2025-06-29 03:35:54,218 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: server-discoverer 2025-06-29 03:35:54,218 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: vmware-discoverer 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: xenserver-discoverer 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: event 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: webhook 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: outofbandmanagement 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: ipmitool 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: nested-cloudstack 2025-06-29 03:35:54,219 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy: redfish 2025-06-29 03:35:54,220 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Could not find module [bootstrap] context as they have no parents. 2025-06-29 03:35:54,220 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [system] context. 2025-06-29 03:35:54,220 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [system] context. 2025-06-29 03:35:54,221 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context.xml] 2025-06-29 03:35:54,221 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context.xml] 2025-06-29 03:35:54,221 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-db-system-context.xml] 2025-06-29 03:35:54,221 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-server-system-context.xml] 2025-06-29 03:35:54,222 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-engine-schema-system-checkers-context.xml] 2025-06-29 03:35:54,222 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:35:54,222 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:35:54,222 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:35:54,222 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:35:55,284 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2025-06-29 03:35:55,285 INFO [c.c.u.d.T.Transaction] (main:[]) (logid:) Is Data Base High Availiability enabled? Ans : false 2025-06-29 03:35:55,285 INFO [c.c.u.d.T.Transaction] (main:[]) (logid:) Using the following URI to connect to cloud database [jdbc:mysql://localhost:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC&scrollTolerantForwardOnly=true]. 2025-06-29 03:35:55,296 DEBUG [c.c.u.d.DriverLoader] (main:[]) (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver 2025-06-29 03:35:55,296 DEBUG [c.c.u.d.T.Transaction] (main:[]) (logid:) Creating datasource for database: cloud with connection pool lib: null 2025-06-29 03:35:55,876 INFO [c.c.u.d.T.Transaction] (main:[]) (logid:) Using the following URI to connect to usage database [jdbc:mysql://localhost:3306/cloud_usage?autoReconnect=true&serverTimezone=UTC&scrollTolerantForwardOnly=true]. 2025-06-29 03:35:55,876 DEBUG [c.c.u.d.DriverLoader] (main:[]) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded. 2025-06-29 03:35:55,876 DEBUG [c.c.u.d.T.Transaction] (main:[]) (logid:) Creating datasource for database: usage with connection pool lib: null 2025-06-29 03:35:55,927 DEBUG [c.c.u.d.DriverLoader] (main:[]) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded. 2025-06-29 03:35:55,927 DEBUG [c.c.u.d.T.Transaction] (main:[]) (logid:) Creating datasource for database: simulator with connection pool lib: null 2025-06-29 03:36:01,066 DEBUG [c.c.u.d.T.Transaction] (main:[]) (logid:) Simulator DB properties are not available. Not initializing simulator DS 2025-06-29 03:36:01,077 DEBUG [c.c.u.d.ConnectionConcierge] (main:[]) (logid:) Registering a database connection for LockController1 2025-06-29 03:36:01,079 INFO [c.c.u.d.Merovingian2] (main:[]) (logid:) Cleaning up locks for 90520738888109 2025-06-29 03:36:01,121 INFO [c.c.u.d.Merovingian2] (main:[]) (logid:) Released 0 locks for 90520738888109 2025-06-29 03:36:01,369 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Running system integrity checker com.cloud.upgrade.DatabaseUpgradeChecker@4c2c44e8 2025-06-29 03:36:01,371 INFO [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) Grabbing lock to check for database upgrade. 2025-06-29 03:36:01,382 DEBUG [c.c.u.c.DBEncryptionUtil] (main:[]) (logid:) Calling to initialize 2025-06-29 03:36:01,382 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2025-06-29 03:36:01,382 DEBUG [c.c.u.c.CloudStackEncryptor] (main:[]) (logid:) Calling to initialize for class com.cloud.utils.crypt.DBEncryptionUtil 2025-06-29 03:36:01,384 DEBUG [c.c.u.c.CloudStackEncryptor] (main:[]) (logid:) Initialized with encryptor : AeadBase64Encryptor 2025-06-29 03:36:01,384 DEBUG [c.c.u.c.DBEncryptionUtil] (main:[]) (logid:) initialized 2025-06-29 03:36:01,385 INFO [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) init = true 2025-06-29 03:36:01,415 DEBUG [c.c.u.d.VersionDaoImpl] (main:[]) (logid:) Checking to see if the database is at a version before it was the version table is created 2025-06-29 03:36:01,439 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Looking for file [ ./engine/schema/dist/systemvm-templates/metadata.ini ] in the classpath. 2025-06-29 03:36:01,439 DEBUG [c.c.u.SystemVmTemplateRegistration] (main:[]) (logid:) Looking for file [ /usr/share/cloudstack-management/templates/systemvm/metadata.ini ] in the classpath. 2025-06-29 03:36:01,464 INFO [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) DB version = 4.21.0.0 Code Version = 4.21.0.0 2025-06-29 03:36:01,464 INFO [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) DB version and code version matches so no upgrade needed. 2025-06-29 03:36:01,469 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:01,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VersionDaoImpl. 2025-06-29 03:36:01,471 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:01,473 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:01,473 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [system] in 7252 ms 2025-06-29 03:36:01,474 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [core] context. 2025-06-29 03:36:01,474 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [core] context. 2025-06-29 03:36:01,474 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-registry-core-context.xml] 2025-06-29 03:36:01,474 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-context.xml] 2025-06-29 03:36:01,474 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-framework-cluster-core-context.xml] 2025-06-29 03:36:01,474 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-framework-jobs-core-context.xml] 2025-06-29 03:36:01,474 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-framework-security-core-context.xml] 2025-06-29 03:36:01,474 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-server-core-managers-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-server-core-misc-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-dedicated-resources-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-internallb-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-baremetal-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-ucs-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-hyperv-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-userdata-cloud-init-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-userdata-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-framework-ipc-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-api-core-context.xml] 2025-06-29 03:36:01,475 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-components-api-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-orchestration-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-schema-core-common-daos-between-management-and-usage-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-schema-core-daos-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-cache-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-services-secondary-storage-controller-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-image-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-datamotion-core-context.xml] 2025-06-29 03:36:01,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-snapshot-core-context.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-volume-core-context.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-engine-storage-object-core-context.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-vmware-core-context.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-nsx-core-managers-context.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:01,477 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:01,478 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:01,478 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [core] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:03,111 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,200 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,270 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,328 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,379 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,401 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,425 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,448 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,461 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,507 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,522 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,550 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,574 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,595 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,612 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:03,635 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:04,119 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from DataCenterIpAddressDaoImpl 2025-06-29 03:36:05,070 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HostTagsDaoImpl 2025-06-29 03:36:05,209 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServiceConfigurationImpl 2025-06-29 03:36:07,550 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ResourceLimitManagerImpl 2025-06-29 03:36:08,023 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:08,056 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Site2 Site Vpn Service Providers Registry] 2025-06-29 03:36:08,143 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:08,153 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Network A C L Service Providers Registry] 2025-06-29 03:36:08,607 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:08,749 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from DeploymentPlanningManagerImpl 2025-06-29 03:36:09,386 INFO [c.c.s.GsonHelper] (main:[]) (logid:) Default Builder inited. 2025-06-29 03:36:09,487 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:09,515 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:09,525 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:09,761 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from IndirectAgentLBServiceImpl 2025-06-29 03:36:09,798 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from PremiumSecondaryStorageManagerImpl 2025-06-29 03:36:10,164 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserDataManagerImpl 2025-06-29 03:36:10,181 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServerImpl 2025-06-29 03:36:10,460 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VolumeApiServiceImpl 2025-06-29 03:36:10,600 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VMSnapshotManagerImpl 2025-06-29 03:36:10,649 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from AsyncJobManagerImpl 2025-06-29 03:36:10,708 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SnapshotManagerImpl 2025-06-29 03:36:10,768 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from DirectDownloadManagerImpl 2025-06-29 03:36:10,851 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:11,014 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VnfTemplateManagerImpl 2025-06-29 03:36:11,029 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from TemplateManagerImpl 2025-06-29 03:36:11,100 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VolumeOrchestrator 2025-06-29 03:36:11,172 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ReconcileCommandServiceImpl 2025-06-29 03:36:11,235 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ClusterManagerImpl 2025-06-29 03:36:11,320 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RoleManagerImpl 2025-06-29 03:36:11,390 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:11,394 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from AnnotationManagerImpl 2025-06-29 03:36:11,412 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ResourceManagerImpl 2025-06-29 03:36:11,436 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KeysManagerImpl 2025-06-29 03:36:11,472 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ConsoleAccessManagerImpl 2025-06-29 03:36:11,512 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ConsoleProxyManagerImpl 2025-06-29 03:36:11,591 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HighAvailabilityManagerExtImpl 2025-06-29 03:36:11,815 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ResourceCleanupServiceImpl 2025-06-29 03:36:11,932 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ClusteredVirtualMachineManagerImpl 2025-06-29 03:36:12,397 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from Ipv6ServiceImpl 2025-06-29 03:36:12,659 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RoutedIpv4ManagerImpl 2025-06-29 03:36:12,724 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VpcVirtualNetworkApplianceManagerImpl 2025-06-29 03:36:12,894 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:12,902 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Vpc Providers Registry] 2025-06-29 03:36:12,903 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VpcManagerImpl 2025-06-29 03:36:12,912 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ProjectManagerImpl 2025-06-29 03:36:12,970 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:12,979 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Firewall Service Providers Registry] 2025-06-29 03:36:12,981 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:12,997 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:12,998 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,029 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Static Nat Service Providers Registry] 2025-06-29 03:36:13,277 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from AutoScaleManagerImpl 2025-06-29 03:36:13,424 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,440 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:13,442 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [InternalLbVm] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:13,455 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,478 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Remote Access V P N Service Provider Registry] 2025-06-29 03:36:13,484 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RemoteAccessVpnManagerImpl 2025-06-29 03:36:13,532 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from IpAddressManagerImpl 2025-06-29 03:36:13,666 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,673 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,673 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [InternalLbVm] in [Internal Load Balancer Element Service Registry] 2025-06-29 03:36:13,674 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NetworkServiceImpl 2025-06-29 03:36:13,765 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VMLeaseManagerImpl 2025-06-29 03:36:13,792 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ConfigurationManagerImpl 2025-06-29 03:36:13,930 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,938 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Network Elements Registry] 2025-06-29 03:36:13,938 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [InternalLbVm] in [Network Elements Registry] 2025-06-29 03:36:13,939 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:13,946 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Dhcp Providers Registry] 2025-06-29 03:36:13,948 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NetworkOrchestrator 2025-06-29 03:36:14,089 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:14,094 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BackupManagerImpl 2025-06-29 03:36:14,232 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VMSchedulerImpl 2025-06-29 03:36:14,342 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserVmManagerImpl 2025-06-29 03:36:14,507 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from StatsCollector 2025-06-29 03:36:14,608 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ImageStoreProviderManagerImpl 2025-06-29 03:36:14,631 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ObjectStoreProviderManagerImpl 2025-06-29 03:36:14,635 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from CapacityManagerImpl 2025-06-29 03:36:14,688 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from AlertManagerImpl 2025-06-29 03:36:14,744 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:14,749 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from CAManagerImpl 2025-06-29 03:36:14,811 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:14,850 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ClusteredAgentManagerImpl 2025-06-29 03:36:15,051 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from StorageManagerImpl 2025-06-29 03:36:15,245 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,262 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from PasswordPolicyImpl 2025-06-29 03:36:15,326 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,339 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,350 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from AccountManagerImpl 2025-06-29 03:36:15,463 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NetworkModelImpl 2025-06-29 03:36:15,528 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,540 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouter] in [Ip Deployers Registry] 2025-06-29 03:36:15,545 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,551 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,552 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,552 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,552 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,553 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,560 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2025-06-29 03:36:15,564 INFO [c.c.u.c.ComponentContext] (main:[]) (logid:) Setup Spring Application context 2025-06-29 03:36:15,566 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2025-06-29 03:36:15,599 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserPasswordResetManagerImpl 2025-06-29 03:36:15,668 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HAManagerImpl 2025-06-29 03:36:15,758 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from OutOfBandManagementServiceImpl 2025-06-29 03:36:16,443 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from QueryManagerImpl 2025-06-29 03:36:16,548 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VirtualNetworkApplianceManagerImpl 2025-06-29 03:36:16,732 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from StorageOrchestrator 2025-06-29 03:36:17,101 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ImageStoreUploadMonitorImpl 2025-06-29 03:36:17,134 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from DiagnosticsServiceImpl 2025-06-29 03:36:17,194 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RollingMaintenanceManagerImpl 2025-06-29 03:36:17,248 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ClusterDrsServiceImpl 2025-06-29 03:36:17,309 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BucketApiServiceImpl 2025-06-29 03:36:17,430 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SharedFSServiceImpl 2025-06-29 03:36:17,572 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ApiServer 2025-06-29 03:36:18,702 DEBUG [o.a.c.f.j.i.JobSerializerHelper] (main:[]) (logid:) Job GSON Builder initialized. 2025-06-29 03:36:26,438 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VmwareManagerImpl 2025-06-29 03:36:26,520 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NsxServiceImpl 2025-06-29 03:36:27,018 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Vm Snapshot Strategies Registry] in [Registry Registry] 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@443b9ebb 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Network Elements Registry] in [Registry Registry] 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@2292e793 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Static Nat Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@4d2bcca8 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Query Selectors Registry] in [Registry Registry] 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@6fc9c0cc 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Network A C L Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@2eb31735 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Vpc Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,019 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@10d8769e 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ha Fence Builders Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@5ddf5118 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ha Investigators Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@7ae0a26 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Pluggable A P I Authenticators Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@1e6bd367 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Template Adapter Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@64fc6470 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Affinity Processors Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@28d37d43 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Kubernetes Service Helper Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@191cf38f 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Gslb Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,020 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@697797e4 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ha Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@406cf37f 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Firewall Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@1ced1fb9 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Network Gurus Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@386b4f7e 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Host Allocators Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@75063bd0 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Dhcp Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@58dd584a 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Snapshot Strategies Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@7c4de811 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Deployment Planners Registry] in [Registry Registry] 2025-06-29 03:36:27,021 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@7b9d1a4 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ip Deployers Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@5bdfae42 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Port Forwarding Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@5d28e108 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Site2 Site Vpn Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@684bfba9 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [User Two Factor Authenticators Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@2459333a 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Event Bus Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@241861bc 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Pod Allocators Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@27d6267e 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Storage Pool Allocators Registry] in [Registry Registry] 2025-06-29 03:36:27,022 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@1760e688 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Resource Discoverers Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@22854f2b 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Data Motion Strategies Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@4db10b9f 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Security Checkers Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@33f2cf82 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Internal Load Balancer Element Service Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@4c131823 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Direct Download Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@74ee07e 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Shared F S Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@5078e308 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Api Checkers Registry] in [Registry Registry] 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@45382749 2025-06-29 03:36:27,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Load Balancing Service Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@1641dfb3 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Secondary Storage Vm Allocators Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@53fc870f 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Hypervisor Gurus Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@4823f4d0 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Backup Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@19544627 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Out Of Band Management Drivers Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@7d412ab5 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [User Password Encoders Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@f6bc75c 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ca Providers Registry] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@1d0590bc 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DataStoreProviderManagerImpl] in [Registry Registry] 2025-06-29 03:36:27,024 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl@4de14c58 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Cluster Drs Algorithm Registry] in [Registry Registry] 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@3c88db1 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Api Acl Checkers Registry] in [Registry Registry] 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@21a09c5a 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Remote Access V P N Service Provider Registry] in [Registry Registry] 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@4a5dc126 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ha Planners Registry] in [Registry Registry] 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@7030b74c 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Console Proxy Allocators Registry] in [Registry Registry] 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@538f45f1 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [User Authenticators Registry] in [Registry Registry] 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry@4dd752e8 2025-06-29 03:36:27,025 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [UcsManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.ucs.manager.UcsManagerImpl@6c96e1d5 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ClusterDrsServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.cluster.ClusterDrsServiceImpl@3a5812dc 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [RoutedIpv4ManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.RoutedIpv4ManagerImpl@520e6089 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ApiDiscoveryServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.discovery.ApiDiscoveryServiceImpl@745d399a 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [APIAuthenticationManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.api.auth.APIAuthenticationManagerImpl@31d11998 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DiagnosticsServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.diagnostics.DiagnosticsServiceImpl@12c1826 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CAManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ca.CAManagerImpl@2ce7d43 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [InternalLbVm] in [Api Commands Registry] 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.element.InternalLoadBalancerElement@701153c1 2025-06-29 03:36:27,026 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ResourceCleanupServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.resource.ResourceCleanupServiceImpl@c509742 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DirectDownloadManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.direct.download.DirectDownloadManagerImpl@443d94ae 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VnfTemplateManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.template.VnfTemplateManagerImpl@26c494d7 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorageBrowserImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.browser.StorageBrowserImpl@1a120e41 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HAManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ha.HAManagerImpl@67fdc1e9 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ProjectRoleManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.ProjectRoleManagerImpl@6da76bb1 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [managementServerMaintenanceManager] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.maintenance.ManagementServerMaintenanceManagerImpl@7cc966a9 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ManagementServerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,027 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.server.ManagementServerImpl@40dd2cfe 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Baremetal Vlan Manager] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.manager.BaremetalVlanManagerImpl@221c097f 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ipv6ServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.Ipv6ServiceImpl@1c8da7e8 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VMScheduleManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.vm.schedule.VMScheduleManagerImpl@4a29520e 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouter] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VpcVirtualRouterElement@3695e4f2 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BaremetalPxeManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.networkservice.BaremetalPxeManagerImpl@48ac4201 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VmwareManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.vmware.manager.VmwareManagerImpl@e0865e7 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BackupManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.backup.BackupManagerImpl@41a6cd9e 2025-06-29 03:36:27,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Baremetal Manager] in [Api Commands Registry] 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.manager.BaremetalManagerImpl@35eb95d3 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [AnnotationManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.annotation.AnnotationManagerImpl@16a89351 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VolumeImportUnmanageManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.volume.VolumeImportUnmanageManagerImpl@6999dca2 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SharedFSServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.sharedfs.SharedFSServiceImpl@4d87553b 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [RoleManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.RoleManagerImpl@ea3491d 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DedicatedResourceManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:27,029 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.dedicated.DedicatedResourceManagerImpl@5d685156 2025-06-29 03:36:27,138 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [cloud-init] in [User Data Providers Registry] 2025-06-29 03:36:27,138 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.userdata.CloudInitUserDataProvider@21304869 2025-06-29 03:36:27,190 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Running system integrity checker org.apache.cloudstack.utils.identity.ManagementServerNode@3c28cd9a 2025-06-29 03:36:27,190 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Running system integrity checker com.cloud.upgrade.DatabaseIntegrityChecker@2b6c3f70 2025-06-29 03:36:27,191 INFO [c.c.u.DatabaseIntegrityChecker] (main:[]) (logid:) Grabbing lock to check for database integrity. 2025-06-29 03:36:27,192 INFO [c.c.u.DatabaseIntegrityChecker] (main:[]) (logid:) Performing database integrity check 2025-06-29 03:36:27,197 DEBUG [c.c.u.DatabaseIntegrityChecker] (main:[]) (logid:) No duplicate hosts with the same local storage found in database 2025-06-29 03:36:27,217 DEBUG [c.c.u.d.VersionDaoImpl] (main:[]) (logid:) Checking to see if the database is at a version before it was the version table is created 2025-06-29 03:36:27,224 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:27,224 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,232 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,239 DEBUG [c.c.a.r.v.VirtualRoutingResource] (main:[]) (logid:) The router.aggregation.command.each.timeout in seconds is set to 600 2025-06-29 03:36:27,239 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,239 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,241 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,241 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PublicIpQuarantineDaoImpl. 2025-06-29 03:36:27,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PhysicalNetworkDaoImpl. 2025-06-29 03:36:27,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainDetailsDaoImpl. 2025-06-29 03:36:27,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RoleDaoImpl. 2025-06-29 03:36:27,243 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostDetailsDaoImpl. 2025-06-29 03:36:27,243 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StoragePoolWorkDaoImpl. 2025-06-29 03:36:27,244 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConfigurationGroupDaoImpl. 2025-06-29 03:36:27,244 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GuestOsDetailsDaoImpl. 2025-06-29 03:36:27,244 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMNetworkMapDaoImpl. 2025-06-29 03:36:27,245 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceCountDaoImpl. 2025-06-29 03:36:27,245 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmProfileDetailsDaoImpl. 2025-06-29 03:36:27,246 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotJoinDaoImpl. 2025-06-29 03:36:27,246 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmDeployAsIsDetailsDaoImpl. 2025-06-29 03:36:27,247 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMSnapshotDetailsDaoImpl. 2025-06-29 03:36:27,247 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PodVlanDaoImpl. 2025-06-29 03:36:27,248 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HypervisorCapabilitiesDaoImpl. 2025-06-29 03:36:27,249 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageSecurityGroupDaoImpl. 2025-06-29 03:36:27,249 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VnfTemplateDetailsDaoImpl. 2025-06-29 03:36:27,249 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PhysicalNetworkIsolationMethodDaoImpl. 2025-06-29 03:36:27,250 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmGroupStatisticsDaoImpl. 2025-06-29 03:36:27,250 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterDetailsDaoImpl. 2025-06-29 03:36:27,250 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UcsBladeDaoImpl. 2025-06-29 03:36:27,251 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HAConfigDaoImpl. 2025-06-29 03:36:27,251 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerStatusDaoImpl. 2025-06-29 03:36:27,251 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserDataDaoImpl. 2025-06-29 03:36:27,251 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainRouterJoinDaoImpl. 2025-06-29 03:36:27,252 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AccountVlanMapDaoImpl. 2025-06-29 03:36:27,252 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SyncQueueDaoImpl. 2025-06-29 03:36:27,252 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ASNumberDaoImpl. 2025-06-29 03:36:27,253 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConfigurationSubGroupDaoImpl. 2025-06-29 03:36:27,253 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StoragePoolAndAccessGroupMapDaoImpl. 2025-06-29 03:36:27,253 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CommandExecLogDaoImpl. 2025-06-29 03:36:27,254 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TemplateJoinDaoImpl. 2025-06-29 03:36:27,254 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OutOfBandManagementDaoImpl. 2025-06-29 03:36:27,254 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmGroupVmMapDaoImpl. 2025-06-29 03:36:27,255 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserStatsLogDaoImpl. 2025-06-29 03:36:27,255 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceIconDaoImpl. 2025-06-29 03:36:27,255 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NicDaoImpl. 2025-06-29 03:36:27,256 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImageStoreJoinDaoImpl. 2025-06-29 03:36:27,256 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotPolicyDaoImpl. 2025-06-29 03:36:27,256 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotZoneDaoImpl. 2025-06-29 03:36:27,257 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcGatewayDaoImpl. 2025-06-29 03:36:27,257 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupRulesDaoImpl. 2025-06-29 03:36:27,257 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkOfferingDaoImpl. 2025-06-29 03:36:27,258 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PlannerHostReservationDaoImpl. 2025-06-29 03:36:27,258 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirewallRulesDaoImpl. 2025-06-29 03:36:27,258 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterVSMMapDaoImpl. 2025-06-29 03:36:27,259 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VlanDetailsDaoImpl. 2025-06-29 03:36:27,259 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcDaoImpl. 2025-06-29 03:36:27,259 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkDetailsDaoImpl. 2025-06-29 03:36:27,260 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BgpPeerNetworkMapDaoImpl. 2025-06-29 03:36:27,260 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AffinityGroupDaoImpl. 2025-06-29 03:36:27,261 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ServiceOfferingJoinDaoImpl. 2025-06-29 03:36:27,261 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DcDetailsDaoImpl. 2025-06-29 03:36:27,261 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerHostPeerDaoImpl. 2025-06-29 03:36:27,261 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageLoadBalancerPolicyDaoImpl. 2025-06-29 03:36:27,262 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirewallRuleDetailsDaoImpl. 2025-06-29 03:36:27,262 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LegacyZoneDaoImpl. 2025-06-29 03:36:27,263 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AsyncJobJoinMapDaoImpl. 2025-06-29 03:36:27,263 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ipv4GuestSubnetNetworkMapDaoImpl. 2025-06-29 03:36:27,263 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LBHealthCheckPolicyDetailsDaoImpl. 2025-06-29 03:36:27,264 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackupScheduleDaoImpl. 2025-06-29 03:36:27,264 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectRoleDaoImpl. 2025-06-29 03:36:27,264 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcGatewayDetailsDaoImpl. 2025-06-29 03:36:27,265 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcOfferingDetailsDaoImpl. 2025-06-29 03:36:27,265 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageJobDaoImpl. 2025-06-29 03:36:27,265 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpnUserDaoImpl. 2025-06-29 03:36:27,265 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LBStickinessPolicyDetailsDaoImpl. 2025-06-29 03:36:27,266 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CiscoNexusVSMDeviceDaoImpl. 2025-06-29 03:36:27,266 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmTemplateDaoImpl. 2025-06-29 03:36:27,266 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcOfferingDaoImpl. 2025-06-29 03:36:27,267 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainRouterDaoImpl. 2025-06-29 03:36:27,267 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectDaoImpl. 2025-06-29 03:36:27,267 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean InstanceGroupVMMapDaoImpl. 2025-06-29 03:36:27,268 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BucketStatisticsDaoImpl. 2025-06-29 03:36:27,268 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeStatsDaoImpl. 2025-06-29 03:36:27,268 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageEventDaoImpl. 2025-06-29 03:36:27,269 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PrimaryDataStoreDaoImpl. 2025-06-29 03:36:27,269 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AsyncJobDaoImpl. 2025-06-29 03:36:27,269 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BgpPeerDaoImpl. 2025-06-29 03:36:27,270 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageBackupDaoImpl. 2025-06-29 03:36:27,270 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageEventDetailsDaoImpl. 2025-06-29 03:36:27,270 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SharedFSJoinDaoImpl. 2025-06-29 03:36:27,271 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectAccountDaoImpl. 2025-06-29 03:36:27,271 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcOfferingServiceMapDaoImpl. 2025-06-29 03:36:27,271 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMTemplateDetailsDaoImpl. 2025-06-29 03:36:27,272 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CounterDaoImpl. 2025-06-29 03:36:27,272 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterVnetDaoImpl. 2025-06-29 03:36:27,273 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeDetailsDaoImpl. 2025-06-29 03:36:27,273 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BaremetalDhcpDaoImpl. 2025-06-29 03:36:27,273 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMTemplateZoneDaoImpl. 2025-06-29 03:36:27,273 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PortProfileDaoImpl. 2025-06-29 03:36:27,274 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageNetworkOfferingDaoImpl. 2025-06-29 03:36:27,274 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLListDetailsDaoImpl. 2025-06-29 03:36:27,274 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterDaoImpl. 2025-06-29 03:36:27,274 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMComputeTagDaoImpl. 2025-06-29 03:36:27,274 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterJoinDaoImpl. 2025-06-29 03:36:27,275 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsagePortForwardingRuleDaoImpl. 2025-06-29 03:36:27,275 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LoadBalancerDaoImpl. 2025-06-29 03:36:27,275 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HighAvailabilityDaoImpl. 2025-06-29 03:36:27,275 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmCloneSettingDaoImpl. 2025-06-29 03:36:27,276 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SslCertDaoImpl. 2025-06-29 03:36:27,276 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirewallRulesCidrsDaoImpl. 2025-06-29 03:36:27,276 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmDiskStatisticsDaoImpl. 2025-06-29 03:36:27,276 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserAccountJoinDaoImpl. 2025-06-29 03:36:27,276 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMTemplateDaoImpl. 2025-06-29 03:36:27,277 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:[]) (logid:) Found parameter routing unique name null 2025-06-29 03:36:27,277 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:[]) (logid:) Use console proxy template : routing 2025-06-29 03:36:27,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackupDaoImpl. 2025-06-29 03:36:27,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmGroupDaoImpl. 2025-06-29 03:36:27,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupWorkDaoImpl. 2025-06-29 03:36:27,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GuestOSHypervisorDaoImpl. 2025-06-29 03:36:27,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LoadBalancerVMMapDaoImpl. 2025-06-29 03:36:27,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenProviderDaoImpl. 2025-06-29 03:36:27,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteVpnConnectionDetailsDaoImpl. 2025-06-29 03:36:27,287 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AsyncJobJournalDaoImpl. 2025-06-29 03:36:27,287 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean InstanceGroupJoinDaoImpl. 2025-06-29 03:36:27,287 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmwareDatacenterZoneMapDaoImpl. 2025-06-29 03:36:27,288 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RouterNetworkDaoImpl. 2025-06-29 03:36:27,288 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteCustomerGatewayDaoImpl. 2025-06-29 03:36:27,289 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean InstanceGroupDaoImpl. 2025-06-29 03:36:27,289 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotScheduleDaoImpl. 2025-06-29 03:36:27,289 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmJoinDaoImpl. 2025-06-29 03:36:27,290 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmwareDatacenterDaoImpl. 2025-06-29 03:36:27,290 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GlobalLoadBalancerLbRuleMapDaoImpl. 2025-06-29 03:36:27,290 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerHostPeerJoinDaoImpl. 2025-06-29 03:36:27,291 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkPermissionDaoImpl. 2025-06-29 03:36:27,291 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageIPAddressDaoImpl. 2025-06-29 03:36:27,291 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLDaoImpl. 2025-06-29 03:36:27,292 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkOfferingServiceMapDaoImpl. 2025-06-29 03:36:27,292 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RouterHealthCheckResultDaoImpl. 2025-06-29 03:36:27,293 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PhysicalNetworkServiceProviderDaoImpl. 2025-06-29 03:36:27,293 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceLimitDaoImpl. 2025-06-29 03:36:27,293 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotDetailsDaoImpl. 2025-06-29 03:36:27,294 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConditionDaoImpl. 2025-06-29 03:36:27,294 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ServiceOfferingDetailsDaoImpl. 2025-06-29 03:36:27,294 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeDaoImpl. 2025-06-29 03:36:27,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AccountDetailsDaoImpl. 2025-06-29 03:36:27,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SSHKeyPairDaoImpl. 2025-06-29 03:36:27,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmDetailsDaoImpl. 2025-06-29 03:36:27,296 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImageStoreDetailsDaoImpl. 2025-06-29 03:36:27,296 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ObjectInDataStoreDaoImpl. 2025-06-29 03:36:27,299 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotDataStoreDaoImpl. 2025-06-29 03:36:27,306 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PrivateIpDaoImpl. 2025-06-29 03:36:27,307 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConsoleProxyDaoImpl. 2025-06-29 03:36:27,307 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DiskOfferingDetailsDaoImpl. 2025-06-29 03:36:27,307 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerHostDaoImpl. 2025-06-29 03:36:27,307 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMRootDiskTagDaoImpl. 2025-06-29 03:36:27,308 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PodVlanMapDaoImpl. 2025-06-29 03:36:27,308 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PrimaryDataStoreDetailsDaoImpl. 2025-06-29 03:36:27,309 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageVpcDaoImpl. 2025-06-29 03:36:27,309 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostDaoImpl. 2025-06-29 03:36:27,309 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AccountJoinDaoImpl. 2025-06-29 03:36:27,310 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenSecurityGroupRuleDaoImpl. 2025-06-29 03:36:27,310 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VirtualRouterProviderDaoImpl. 2025-06-29 03:36:27,310 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectRolePermissionsDaoImpl. 2025-06-29 03:36:27,311 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserAccountDaoImpl. 2025-06-29 03:36:27,311 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackupOfferingDaoImpl. 2025-06-29 03:36:27,311 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DetailsDao. 2025-06-29 03:36:27,312 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkRuleConfigDaoImpl. 2025-06-29 03:36:27,312 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PhysicalNetworkTagDaoImpl. 2025-06-29 03:36:27,312 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean IPAddressDaoImpl. 2025-06-29 03:36:27,313 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PassphraseDaoImpl. 2025-06-29 03:36:27,313 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainVlanMapDaoImpl. 2025-06-29 03:36:27,313 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirewallRulesDcidrsDaoImpl. 2025-06-29 03:36:27,314 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostGpuGroupsDaoImpl. 2025-06-29 03:36:27,314 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DirectDownloadCertificateHostMapDaoImpl. 2025-06-29 03:36:27,314 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteVpnGatewayDetailsDaoImpl. 2025-06-29 03:36:27,315 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserStatisticsDaoImpl. 2025-06-29 03:36:27,315 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TemplateDataStoreDaoImpl. 2025-06-29 03:36:27,317 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NsxProviderDaoImpl. 2025-06-29 03:36:27,318 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterIpAddressDaoImpl. 2025-06-29 03:36:27,318 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteCustomerGatewayDetailsDaoImpl. 2025-06-29 03:36:27,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMTemplatePoolDaoImpl. 2025-06-29 03:36:27,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmGroupPolicyMapDaoImpl. 2025-06-29 03:36:27,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EngineClusterDaoImpl. 2025-06-29 03:36:27,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecondaryStorageVmDaoImpl. 2025-06-29 03:36:27,320 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImageStoreDaoImpl. 2025-06-29 03:36:27,321 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AffinityGroupDomainMapDaoImpl. 2025-06-29 03:36:27,322 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UcsManagerDaoImpl. 2025-06-29 03:36:27,322 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ASNumberRangeDaoImpl. 2025-06-29 03:36:27,322 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CapacityDaoImpl. 2025-06-29 03:36:27,323 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EventDaoImpl. 2025-06-29 03:36:27,323 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RemoteAccessVpnDetailsDaoImpl. 2025-06-29 03:36:27,323 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterGuestIpv6PrefixDaoImpl. 2025-06-29 03:36:27,324 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RegionDaoImpl. 2025-06-29 03:36:27,324 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkServiceMapDaoImpl. 2025-06-29 03:36:27,325 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMReservationDaoImpl. 2025-06-29 03:36:27,325 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupRuleDaoImpl. 2025-06-29 03:36:27,325 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ReconcileCommandDaoImpl. 2025-06-29 03:36:27,326 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BgpPeerDetailsDaoImpl. 2025-06-29 03:36:27,326 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BaremetalRctDaoImpl. 2025-06-29 03:36:27,326 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostTransferMapDaoImpl. 2025-06-29 03:36:27,326 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ItWorkDaoImpl. 2025-06-29 03:36:27,326 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcOfferingJoinDaoImpl. 2025-06-29 03:36:27,327 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImageStoreObjectDownloadDaoImpl. 2025-06-29 03:36:27,329 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupJoinDaoImpl. 2025-06-29 03:36:27,329 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GlobalLoadBalancerDaoImpl. 2025-06-29 03:36:27,329 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LoadBalancerCertMapDaoImpl. 2025-06-29 03:36:27,330 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DirectDownloadCertificateDaoImpl. 2025-06-29 03:36:27,330 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScalePolicyConditionMapDaoImpl. 2025-06-29 03:36:27,330 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BucketDaoImpl. 2025-06-29 03:36:27,334 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmStatsDaoImpl. 2025-06-29 03:36:27,335 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeDataStoreDaoImpl. 2025-06-29 03:36:27,342 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectInvitationJoinDaoImpl. 2025-06-29 03:36:27,342 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LBStickinessPolicyDaoImpl. 2025-06-29 03:36:27,342 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectJoinDaoImpl. 2025-06-29 03:36:27,343 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VlanDaoImpl. 2025-06-29 03:36:27,343 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StaticRouteDaoImpl. 2025-06-29 03:36:27,344 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostTagsDaoImpl. 2025-06-29 03:36:27,344 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageVolumeDaoImpl. 2025-06-29 03:36:27,344 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkExternalLoadBalancerDaoImpl. 2025-06-29 03:36:27,344 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DiskOfferingDaoImpl. 2025-06-29 03:36:27,345 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkAccountDaoImpl. 2025-06-29 03:36:27,345 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VsphereStoragePolicyDaoImpl. 2025-06-29 03:36:27,345 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageDaoImpl. 2025-06-29 03:36:27,345 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AccountDaoImpl. 2025-06-29 03:36:27,346 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserDaoImpl. 2025-06-29 03:36:27,346 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceTagJoinDaoImpl. 2025-06-29 03:36:27,346 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvsProviderDaoImpl. 2025-06-29 03:36:27,346 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectInvitationDaoImpl. 2025-06-29 03:36:27,347 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainDaoImpl. 2025-06-29 03:36:27,347 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CrlDaoImpl. 2025-06-29 03:36:27,347 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TemplateDeployAsIsDetailsDaoImpl. 2025-06-29 03:36:27,348 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserDetailsDaoImpl. 2025-06-29 03:36:27,348 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EngineDataCenterDaoImpl. 2025-06-29 03:36:27,349 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageVPNUserDaoImpl. 2025-06-29 03:36:27,349 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLItemDaoImpl. 2025-06-29 03:36:27,349 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMScheduleDaoImpl. 2025-06-29 03:36:27,349 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmDaoImpl. 2025-06-29 03:36:27,350 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkExternalFirewallDaoImpl. 2025-06-29 03:36:27,350 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostPodDaoImpl. 2025-06-29 03:36:27,350 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AccountGuestVlanMapDaoImpl. 2025-06-29 03:36:27,350 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteVpnGatewayDaoImpl. 2025-06-29 03:36:27,351 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ipv6GuestPrefixSubnetNetworkMapDaoImpl. 2025-06-29 03:36:27,351 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmGroupDetailsDaoImpl. 2025-06-29 03:36:27,351 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SyncQueueItemDaoImpl. 2025-06-29 03:36:27,351 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RemoteAccessVpnDaoImpl. 2025-06-29 03:36:27,352 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AffinityGroupVMMapDaoImpl. 2025-06-29 03:36:27,352 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BaremetalPxeDaoImpl. 2025-06-29 03:36:27,352 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageVmDiskDaoImpl. 2025-06-29 03:36:27,352 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleVmProfileDaoImpl. 2025-06-29 03:36:27,353 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcServiceMapDaoImpl. 2025-06-29 03:36:27,353 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotDaoImpl. 2025-06-29 03:36:27,353 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EventJoinDaoImpl. 2025-06-29 03:36:27,354 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterDetailsDaoImpl. 2025-06-29 03:36:27,354 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLItemCidrsDaoImpl. 2025-06-29 03:36:27,355 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMScheduledJobDaoImpl. 2025-06-29 03:36:27,355 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLItemDetailsDaoImpl. 2025-06-29 03:36:27,356 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ObjectStoreDetailsDaoImpl. 2025-06-29 03:36:27,356 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NicIpAliasDaoImpl. 2025-06-29 03:36:27,356 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterLinkLocalIpAddressDaoImpl. 2025-06-29 03:36:27,596 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerJoinDaoImpl. 2025-06-29 03:36:27,597 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AlertDaoImpl. 2025-06-29 03:36:27,597 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ReservationDaoImpl. 2025-06-29 03:36:27,597 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VnfTemplateNicDaoImpl. 2025-06-29 03:36:27,598 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecondaryStorageHeuristicDaoImpl. 2025-06-29 03:36:27,598 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkOfferingDetailsDaoImpl. 2025-06-29 03:36:27,598 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ObjectStoreDaoImpl. 2025-06-29 03:36:27,599 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AnnotationDaoImpl. 2025-06-29 03:36:27,599 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AffinityGroupJoinDaoImpl. 2025-06-29 03:36:27,600 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMSnapshotDaoImpl. 2025-06-29 03:36:27,600 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmRulesetLogDaoImpl. 2025-06-29 03:36:27,600 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EngineHostDaoImpl. 2025-06-29 03:36:27,601 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StoragePoolJoinDaoImpl. 2025-06-29 03:36:27,601 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NicExtraDhcpOptionDaoImpl. 2025-06-29 03:36:27,601 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterDrsPlanMigrationDaoImpl. 2025-06-29 03:36:27,601 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DedicatedResourceDaoImpl. 2025-06-29 03:36:27,602 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotPolicyDetailsDaoImpl. 2025-06-29 03:36:27,602 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupDaoImpl. 2025-06-29 03:36:27,603 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GuestOSCategoryDaoImpl. 2025-06-29 03:36:27,603 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean MonitoringServiceDaoImpl. 2025-06-29 03:36:27,603 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StoragePoolHostDaoImpl. 2025-06-29 03:36:27,603 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostJoinDaoImpl. 2025-06-29 03:36:27,603 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeReservationDaoImpl. 2025-06-29 03:36:27,604 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LBHealthCheckPolicyDaoImpl. 2025-06-29 03:36:27,604 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterDaoImpl. 2025-06-29 03:36:27,604 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageNetworkIpAddressDaoImpl. 2025-06-29 03:36:27,605 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserIpAddressDetailsDaoImpl. 2025-06-29 03:36:27,605 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PortableIpDaoImpl. 2025-06-29 03:36:27,605 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmWorkJobDaoImpl. 2025-06-29 03:36:27,606 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserIpv6AddressDaoImpl. 2025-06-29 03:36:27,606 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostTagsDaoImpl. 2025-06-29 03:36:27,606 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteVpnConnectionDaoImpl. 2025-06-29 03:36:27,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageNetworksDaoImpl. 2025-06-29 03:36:27,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageStorageDaoImpl. 2025-06-29 03:36:27,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageNetworkDaoImpl. 2025-06-29 03:36:27,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcDaoImpl. 2025-06-29 03:36:27,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StoragePoolTagsDaoImpl. 2025-06-29 03:36:27,608 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DiskOfferingJoinDaoImpl. 2025-06-29 03:36:27,608 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMInstanceDaoImpl. 2025-06-29 03:36:27,608 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeJoinDaoImpl. 2025-06-29 03:36:27,608 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMEntityDaoImpl. 2025-06-29 03:36:27,608 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SharedFSDaoImpl. 2025-06-29 03:36:27,609 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean InlineLoadBalancerNicMapDaoImpl. 2025-06-29 03:36:27,609 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PhysicalNetworkTrafficTypeDaoImpl. 2025-06-29 03:36:27,609 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkOfferingJoinDaoImpl. 2025-06-29 03:36:27,609 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LaunchPermissionDaoImpl. 2025-06-29 03:36:27,609 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScalePolicyDaoImpl. 2025-06-29 03:36:27,610 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenGuestNetworkIpAddressDaoImpl. 2025-06-29 03:36:27,610 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkDaoImpl. 2025-06-29 03:36:27,610 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PortForwardingRulesDaoImpl. 2025-06-29 03:36:27,610 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkDaoImpl. 2025-06-29 03:36:27,611 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ExternalFirewallDeviceDaoImpl. 2025-06-29 03:36:27,611 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ServiceOfferingDaoImpl. 2025-06-29 03:36:27,611 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApplicationLoadBalancerRuleDaoImpl. 2025-06-29 03:36:27,611 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceTagsDaoImpl. 2025-06-29 03:36:27,611 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ExternalLoadBalancerDeviceDaoImpl. 2025-06-29 03:36:27,612 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NicSecondaryIpDaoImpl. 2025-06-29 03:36:27,612 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectAccountJoinDaoImpl. 2025-06-29 03:36:27,612 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VGPUTypesDaoImpl. 2025-06-29 03:36:27,612 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AsyncJobJoinDaoImpl. 2025-06-29 03:36:27,612 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PortableIpRangeDaoImpl. 2025-06-29 03:36:27,613 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcDetailsDaoImpl. 2025-06-29 03:36:27,613 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainJoinDaoImpl. 2025-06-29 03:36:27,613 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkDomainDaoImpl. 2025-06-29 03:36:27,613 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RolePermissionsDaoImpl. 2025-06-29 03:36:27,613 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageNetworkIpRangeDaoImpl. 2025-06-29 03:36:27,614 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupVMMapDaoImpl. 2025-06-29 03:36:27,614 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NicDetailsDaoImpl. 2025-06-29 03:36:27,614 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostDetailsDaoImpl. 2025-06-29 03:36:27,614 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkOpDaoImpl. 2025-06-29 03:36:27,614 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConsoleSessionDaoImpl. 2025-06-29 03:36:27,615 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EngineHostPodDaoImpl. 2025-06-29 03:36:27,615 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterDrsPlanDaoImpl. 2025-06-29 03:36:27,615 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataCenterIpv4GuestSubnetDaoImpl. 2025-06-29 03:36:27,615 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackupRepositoryDaoImpl. 2025-06-29 03:36:27,615 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UploadDaoImpl. 2025-06-29 03:36:27,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KeystoreDaoImpl. 2025-06-29 03:36:27,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CertificateDaoImpl. 2025-06-29 03:36:27,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GuestOSDaoImpl. 2025-06-29 03:36:27,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OpRouterMonitorServiceDaoImpl. 2025-06-29 03:36:27,616 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageVMInstanceDaoImpl. 2025-06-29 03:36:27,617 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerNode. 2025-06-29 03:36:27,617 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DatabaseIntegrityChecker. 2025-06-29 03:36:27,617 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConfigurationServerImpl. 2025-06-29 03:36:27,622 DEBUG [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Looks like system account already exists 2025-06-29 03:36:27,626 DEBUG [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Looks like system user already exists 2025-06-29 03:36:27,632 DEBUG [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Looks like admin account already exists 2025-06-29 03:36:27,642 DEBUG [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Looks like admin user already exists 2025-06-29 03:36:27,671 INFO [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Processing updateKeyPairs 2025-06-29 03:36:27,672 INFO [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Keypairs already in database, updating local copy 2025-06-29 03:36:27,677 INFO [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Trying to copy private keys to hosts 2025-06-29 03:36:27,684 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Looking for scripts/vm/systemvm/injectkeys.sh in the classpath 2025-06-29 03:36:27,684 DEBUG [c.c.u.s.Script] (main:[]) (logid:) System resource: file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh 2025-06-29 03:36:27,685 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Absolute path = /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh 2025-06-29 03:36:27,687 DEBUG [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Executing command [/bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa ]. 2025-06-29 03:36:27,705 DEBUG [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) Successfully executed process [4064579] for command [/bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa ]. 2025-06-29 03:36:27,705 INFO [c.c.s.ConfigurationServerImpl] (main:[]) (logid:) The script injectkeys.sh was run with result : null 2025-06-29 03:36:27,724 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AsyncJobManagerImpl. 2025-06-29 03:36:27,724 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2025-06-29 03:36:27,725 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Start AsyncJobManager API executor thread pool in size 125 2025-06-29 03:36:27,725 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Start AsyncJobManager Work executor thread pool in size 166 2025-06-29 03:36:27,726 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AsyncJobMonitor. 2025-06-29 03:36:27,726 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Build message handler cache for org.apache.cloudstack.framework.jobs.impl.AsyncJobMonitor 2025-06-29 03:36:27,727 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Add message handler org.apache.cloudstack.framework.jobs.impl.AsyncJobMonitor.onJobHeartbeatNotify to cache 2025-06-29 03:36:27,727 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Done building message handler cache for org.apache.cloudstack.framework.jobs.impl.AsyncJobMonitor 2025-06-29 03:36:27,727 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcManagerImpl. 2025-06-29 03:36:27,889 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QueryManagerImpl. 2025-06-29 03:36:27,889 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SharedFSServiceImpl. 2025-06-29 03:36:27,891 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TaggedResourceManagerImpl. 2025-06-29 03:36:27,892 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AnnotationManagerImpl. 2025-06-29 03:36:27,892 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectManagerImpl. 2025-06-29 03:36:27,898 DEBUG [o.a.c.u.m.SMTPMailSender] (main:[]) (logid:) Unable to instantiate SMTP mail session due to empty or null host. 2025-06-29 03:36:27,898 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceCleanupServiceImpl. 2025-06-29 03:36:27,899 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VnfTemplateManagerImpl. 2025-06-29 03:36:27,899 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkModelImpl. 2025-06-29 03:36:27,912 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Network Model is configured. 2025-06-29 03:36:27,912 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApplicationLoadBalancerManagerImpl. 2025-06-29 03:36:27,913 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LoadBalancingRulesManagerImpl. 2025-06-29 03:36:27,913 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmManagerImpl. 2025-06-29 03:36:27,920 INFO [c.c.v.UserVmManagerImpl] (main:[]) (logid:) User VM Manager is configured. 2025-06-29 03:36:27,920 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserDataManagerImpl. 2025-06-29 03:36:27,920 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ipv6AddressManagerImpl. 2025-06-29 03:36:27,920 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ExternalNetworkDeviceManagerImpl. 2025-06-29 03:36:27,920 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,921 INFO [c.c.s.StorageManagerImpl] (main:[]) (logid:) pool.acquisition.wait.seconds is configured as 1800 seconds 2025-06-29 03:36:27,922 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener StoragePoolMonitor with id 1 2025-06-29 03:36:27,932 INFO [c.c.s.StorageManagerImpl] (main:[]) (logid:) Storage cleanup enabled: true, interval: 86400, delay: 86400, template cleanup enabled: true 2025-06-29 03:36:27,936 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener LocalStoragePoolListener with id 2 2025-06-29 03:36:27,938 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean IpAddressManagerImpl. 2025-06-29 03:36:27,942 INFO [c.c.n.IpAddressManagerImpl] (main:[]) (logid:) IPAddress Manager is configured. 2025-06-29 03:36:27,942 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApiServer. 2025-06-29 03:36:27,942 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Build message handler cache for com.cloud.api.ApiServer 2025-06-29 03:36:27,942 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Add message handler com.cloud.api.ApiServer.handleAsyncJobPublishEvent to cache 2025-06-29 03:36:27,942 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Done building message handler cache for com.cloud.api.ApiServer 2025-06-29 03:36:27,942 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConsoleAccessManagerImpl. 2025-06-29 03:36:27,942 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ReconcileCommandServiceImpl. 2025-06-29 03:36:27,944 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OCFS2ManagerImpl. 2025-06-29 03:36:27,944 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HypervisorGuruManagerImpl. 2025-06-29 03:36:27,944 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLServiceImpl. 2025-06-29 03:36:27,944 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageOrchestrator. 2025-06-29 03:36:27,945 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Baremetal Manager. 2025-06-29 03:36:27,946 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UsageServiceImpl. 2025-06-29 03:36:27,947 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BaremetalKickStartServiceImpl. 2025-06-29 03:36:27,947 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:27,947 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TemplateManagerImpl. 2025-06-29 03:36:27,949 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HAManagerImpl. 2025-06-29 03:36:27,964 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Background Poll Manager received task: HAManagerBgPollTask 2025-06-29 03:36:27,964 DEBUG [o.a.c.h.HAManagerImpl] (main:[]) (logid:) HA manager has been configured. 2025-06-29 03:36:27,964 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkOrchestrator. 2025-06-29 03:36:27,995 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener NetworkOrchestrator with id 3 2025-06-29 03:36:27,996 INFO [o.a.c.e.o.NetworkOrchestrator] (main:[]) (logid:) Network Manager is configured. 2025-06-29 03:36:27,996 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean APIAuthenticationManagerImpl. 2025-06-29 03:36:27,996 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BucketApiServiceImpl. 2025-06-29 03:36:27,996 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmwareManagerImpl. 2025-06-29 03:36:27,996 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) Configure VmwareManagerImpl, manager name: VmwareManagerImpl 2025-06-29 03:36:27,998 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) VmwareManagerImpl config - instance.name: VM 2025-06-29 03:36:28,001 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) VmwareManagerImpl config - _mountParent: /var/cloudstack/mnt/VM 2025-06-29 03:36:28,013 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) VmwareManagerImpl config - vmware.vcenter.session.timeout: 1200000 2025-06-29 03:36:28,018 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) Additional VNC port allocation range is settled at 50000 to 51000 2025-06-29 03:36:28,018 INFO [c.c.h.v.m.VmwareStorageManagerImpl] (main:[]) (logid:) Configure VmwareStorageManagerImpl 2025-06-29 03:36:28,018 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener VmwareManagerImpl with id 4 2025-06-29 03:36:28,018 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) VmwareManagerImpl has been successfully configured 2025-06-29 03:36:28,018 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageBrowserImpl. 2025-06-29 03:36:28,018 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserPasswordResetManagerImpl. 2025-06-29 03:36:28,025 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DeploymentPlanningManagerImpl. 2025-06-29 03:36:28,026 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener DeploymentPlanningManagerImpl with id 5 2025-06-29 03:36:28,029 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CapacityManagerImpl. 2025-06-29 03:36:28,032 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener StorageCapacityListener with id 6 2025-06-29 03:36:28,032 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener ComputeCapacityListener with id 7 2025-06-29 03:36:28,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Baremetal Vlan Manager. 2025-06-29 03:36:28,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMLeaseManagerImpl. 2025-06-29 03:36:28,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkUsageManagerImpl. 2025-06-29 03:36:28,041 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener DirectNetworkStatsListener with id 8 2025-06-29 03:36:28,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Volume Manager. 2025-06-29 03:36:28,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceIconManagerImpl. 2025-06-29 03:36:28,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceManagerImpl. 2025-06-29 03:36:28,044 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotManagerImpl. 2025-06-29 03:36:28,056 INFO [c.c.s.s.SnapshotManagerImpl] (main:[]) (logid:) Snapshot Manager is configured. 2025-06-29 03:36:28,056 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainManagerImpl. 2025-06-29 03:36:28,056 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RoleManagerImpl. 2025-06-29 03:36:28,056 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackgroundPollManagerImpl. 2025-06-29 03:36:28,056 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageNetworkManagerImpl. 2025-06-29 03:36:28,056 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StatsCollector. 2025-06-29 03:36:28,056 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterDrsServiceImpl. 2025-06-29 03:36:28,057 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PremiumSecondaryStorageManagerImpl. 2025-06-29 03:36:28,057 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (main:[]) (logid:) Start configuring secondary storage vm manager : PremiumSecondaryStorageManagerImpl 2025-06-29 03:36:28,062 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener SecondaryStorageListener with id 9 2025-06-29 03:36:28,085 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (main:[]) (logid:) Secondary storage VM manager [PremiumSecondaryStorageManagerImpl] was configured. 2025-06-29 03:36:28,090 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConfigurationManagerImpl. 2025-06-29 03:36:28,094 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusteredAgentManagerImpl. 2025-06-29 03:36:28,094 INFO [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Configuring ClusterAgentManagerImpl. management server node id(msid): 90520738888109 2025-06-29 03:36:28,098 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) register cluster listener class com.cloud.agent.manager.ClusteredAgentManagerImpl 2025-06-29 03:36:28,101 INFO [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Ping Timeout is 2.5. 2025-06-29 03:36:28,101 INFO [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Configuring AgentManagerImpl. management server node id(msid): 90520738888109. 2025-06-29 03:36:28,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener BehindOnPingListener with id 10 2025-06-29 03:36:28,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener SetHostParamsListener with id 11 2025-06-29 03:36:28,123 INFO [o.a.c.m.ManagementServerMaintenanceManagerImpl] (main:[]) (logid:) Register management server maintenance listener class com.cloud.agent.manager.ClusteredAgentManagerImpl 2025-06-29 03:36:28,132 INFO [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Listening on 8250 with 5 workers. 2025-06-29 03:36:28,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Created DirectAgentAttache pool with size: 500. 2025-06-29 03:36:28,139 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RollingMaintenanceManagerImpl. 2025-06-29 03:36:28,139 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SnapshotSchedulerImpl. 2025-06-29 03:36:28,142 INFO [c.c.s.s.SnapshotSchedulerImpl] (main:[]) (logid:) Snapshot Scheduler is configured. 2025-06-29 03:36:28,142 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BareMetalPingServiceImpl. 2025-06-29 03:36:28,142 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CAManagerImpl. 2025-06-29 03:36:28,143 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Background Poll Manager received task: CABackgroundTask 2025-06-29 03:36:28,143 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImageStoreUploadMonitorImpl. 2025-06-29 03:36:28,147 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VirtualNetworkApplianceManagerImpl. 2025-06-29 03:36:28,150 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (main:[]) (logid:) Router configurations: ramsize=256 2025-06-29 03:36:28,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener SshKeysDistriMonitor with id 12 2025-06-29 03:36:28,165 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener VirtualNetworkApplianceManagerImpl with id 13 2025-06-29 03:36:28,165 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (main:[]) (logid:) DomainRouterManager is configured. 2025-06-29 03:36:28,165 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DirectDownloadManagerImpl. 2025-06-29 03:36:28,165 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:28,167 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean InternalLoadBalancerVMManagerImpl. 2025-06-29 03:36:28,178 INFO [o.a.c.n.l.InternalLoadBalancerVMManagerImpl] (main:[]) (logid:) InternalLoadBalancerVMManagerImpl has been configured 2025-06-29 03:36:28,178 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterFenceManagerImpl. 2025-06-29 03:36:28,178 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) register cluster listener class com.cloud.cluster.ClusterFenceManagerImpl 2025-06-29 03:36:28,179 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceLimitManagerImpl. 2025-06-29 03:36:28,245 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ResourceMetaDataManagerImpl. 2025-06-29 03:36:28,245 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConsoleProxyManagerImpl. 2025-06-29 03:36:28,245 INFO [c.c.c.ConsoleProxyManagerImpl] (main:[]) (logid:) Start configuring console proxy manager : ConsoleProxyManagerImpl 2025-06-29 03:36:28,246 INFO [c.c.c.ConsoleProxyManagerImpl] (main:[]) (logid:) Console proxy max session soft limit : 50 2025-06-29 03:36:28,246 INFO [c.c.c.ConsoleProxyManagerImpl] (main:[]) (logid:) Console proxy standby capacity : 10 2025-06-29 03:36:28,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener ConsoleProxyListener with id 14 2025-06-29 03:36:28,265 INFO [c.c.c.ConsoleProxyManagerImpl] (main:[]) (logid:) Console Proxy Manager is configured. 2025-06-29 03:36:28,265 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:28,265 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcVirtualNetworkApplianceManagerImpl. 2025-06-29 03:36:28,268 INFO [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (main:[]) (logid:) Router configurations: ramsize=256 2025-06-29 03:36:28,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener SshKeysDistriMonitor with id 15 2025-06-29 03:36:28,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener VpcVirtualNetworkApplianceManagerImpl with id 16 2025-06-29 03:36:28,278 INFO [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (main:[]) (logid:) DomainRouterManager is configured. 2025-06-29 03:36:28,278 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:28,278 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2025-06-29 03:36:28,279 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackupRepositoryServiceImpl. 2025-06-29 03:36:28,279 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BaremetalPxeManagerImpl. 2025-06-29 03:36:28,279 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Site2SiteVpnManagerImpl. 2025-06-29 03:36:28,279 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImageStoreServiceImpl. 2025-06-29 03:36:28,279 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DataStoreProviderManagerImpl. 2025-06-29 03:36:28,280 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean com.cloud.network.security.SecurityGroupManagerImpl2. 2025-06-29 03:36:28,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener SecurityGroupListener with id 17 2025-06-29 03:36:28,285 INFO [c.c.n.s.SecurityGroupManagerImpl2] (main:[]) (logid:) SecurityGroupManager: num worker threads=50, time between cleanups=120 global lock timeout=300 2025-06-29 03:36:28,288 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:28,317 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Background Poll Manager received task: OutOfBandManagementPowerStatePollTask 2025-06-29 03:36:28,318 INFO [o.a.c.o.OutOfBandManagementServiceImpl] (main:[]) (logid:) Starting out-of-band management background sync executor with thread pool-size=50 2025-06-29 03:36:28,318 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:28,318 INFO [c.c.n.l.LBHealthCheckManagerImpl] (main:[]) (logid:) Configuring LBHealthCheck Manager null 2025-06-29 03:36:28,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VolumeApiServiceImpl. 2025-06-29 03:36:28,334 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AccountManagerImpl. 2025-06-29 03:36:28,341 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMSchedulerImpl. 2025-06-29 03:36:28,341 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMScheduleManagerImpl. 2025-06-29 03:36:28,341 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AutoScaleManagerImpl. 2025-06-29 03:36:28,341 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkACLManagerImpl. 2025-06-29 03:36:28,342 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean managementServerMaintenanceManager. 2025-06-29 03:36:28,342 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AlertManagerImpl. 2025-06-29 03:36:28,342 DEBUG [o.a.c.u.m.SMTPMailSender] (main:[]) (logid:) Unable to instantiate SMTP mail session due to empty or null host. 2025-06-29 03:36:28,351 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirewallManagerImpl. 2025-06-29 03:36:28,353 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RulesManagerImpl. 2025-06-29 03:36:28,353 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkServiceImpl. 2025-06-29 03:36:28,353 INFO [c.c.n.NetworkServiceImpl] (main:[]) (logid:) Network Service is configured. 2025-06-29 03:36:28,353 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusteredVirtualMachineManagerImpl. 2025-06-29 03:36:28,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener ClusteredVirtualMachineManagerImpl with id 18 2025-06-29 03:36:28,355 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Build message handler cache for com.cloud.vm.ClusteredVirtualMachineManagerImpl 2025-06-29 03:36:28,355 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Add message handler com.cloud.vm.ClusteredVirtualMachineManagerImpl.HandlePowerStateReport to cache 2025-06-29 03:36:28,355 INFO [o.a.c.f.m.MessageDispatcher] (main:[]) (logid:) Done building message handler cache for com.cloud.vm.ClusteredVirtualMachineManagerImpl 2025-06-29 03:36:28,366 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) register cluster listener class com.cloud.vm.ClusteredVirtualMachineManagerImpl 2025-06-29 03:36:28,366 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DownloadMonitorImpl. 2025-06-29 03:36:28,366 WARN [c.c.s.d.DownloadMonitorImpl] (main:[]) (logid:) Only realhostip.com ssl cert is supported, ignoring self-signed and other certs 2025-06-29 03:36:28,379 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener DownloadListener with id 19 2025-06-29 03:36:28,379 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean EventDistributorImpl. 2025-06-29 03:36:28,379 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectRoleManagerImpl. 2025-06-29 03:36:28,379 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RemoteAccessVpnManagerImpl. 2025-06-29 03:36:28,398 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BackupManagerImpl. 2025-06-29 03:36:28,399 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Background Poll Manager received task: BackupSyncTask 2025-06-29 03:36:28,406 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DiagnosticsServiceImpl. 2025-06-29 03:36:28,408 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Background Poll Manager received task: GCBackgroundTask 2025-06-29 03:36:28,408 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SyncQueueManagerImpl. 2025-06-29 03:36:28,408 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UploadMonitorImpl. 2025-06-29 03:36:28,412 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener UploadListener with id 20 2025-06-29 03:36:28,412 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMSnapshotManagerImpl. 2025-06-29 03:36:28,414 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HighAvailabilityManagerExtImpl. 2025-06-29 03:36:28,438 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KeystoreManagerImpl. 2025-06-29 03:36:28,438 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterManagerImpl. 2025-06-29 03:36:28,439 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Start configuring cluster manager : ClusterManagerImpl 2025-06-29 03:36:28,439 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2025-06-29 03:36:28,439 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Cluster node IP : 172.16.20.40 2025-06-29 03:36:28,452 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Trying to connect to 172.16.20.40 2025-06-29 03:36:28,459 ERROR [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Unable to ping management server at 172.16.20.40:9090 due to ConnectException 2025-06-29 03:36:28,464 DEBUG [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Unable to ping management server at 172.16.20.40:9090 due to ConnectException java.net.ConnectException: Connection refused at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:579) at java.base/sun.nio.ch.Net.connect(Net.java:586) at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853) at com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1268) at com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1237) at com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1318) at com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1193) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:121) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:163) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:116) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:53) at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:174) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:143) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:274) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:279) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:262) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:131) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79) at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:58) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:62) at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:53) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:204) at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:115) 2025-06-29 03:36:28,465 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Detected that another management node with the same IP 172.16.20.40 is considered as running in DB, however it is not pingable, we will continue cluster initialization with this management server node 2025-06-29 03:36:28,465 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Cluster manager is configured. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DumpRegistry. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmWorkJobDispatcher. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApiDiscoveryServiceImpl. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcVirtualRouter. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RoutedIpv4ManagerImpl. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApiAsyncJobDispatcher. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VmWorkJobWakeupDispatcher. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean InternalLbVm. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ipv6ServiceImpl. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterBasedAgentLoadBalancerPlanner. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CloudZonesStartupProcessor. 2025-06-29 03:36:28,465 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean IndirectAgentLBServiceImpl. 2025-06-29 03:36:28,466 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean cloud-init. 2025-06-29 03:36:28,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BASIC. 2025-06-29 03:36:28,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterServiceServletAdapter. 2025-06-29 03:36:28,467 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Basic. 2025-06-29 03:36:28,470 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementServerImpl. 2025-06-29 03:36:28,474 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DataCenterDetailsDaoImpl] in [Scoped Config Storage Registry] 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.dc.dao.DataCenterDetailsDaoImpl@36885319 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ImageStoreDetailsDaoImpl] in [Scoped Config Storage Registry] 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.db.ImageStoreDetailsDaoImpl@18baee88 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [AccountDetailsDaoImpl] in [Scoped Config Storage Registry] 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.user.AccountDetailsDaoImpl@75cfb958 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DetailsDao] in [Scoped Config Storage Registry] 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.storage.dao.StoragePoolDetailsDaoImpl@3c5044fa 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DomainDetailsDaoImpl] in [Scoped Config Storage Registry] 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.domain.dao.DomainDetailsDaoImpl@5687e92d 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ClusterDetailsDaoImpl] in [Scoped Config Storage Registry] 2025-06-29 03:36:28,475 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.dc.ClusterDetailsDaoImpl@550574cb 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [core] in 27001 ms 2025-06-29 03:36:28,476 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [allocator] context. 2025-06-29 03:36:28,476 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [allocator] context. 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-allocator-context.xml] 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:28,476 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:28,590 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:28,590 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [allocator] in 115 ms 2025-06-29 03:36:28,591 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [host-allocator-random] context. 2025-06-29 03:36:28,591 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [host-allocator-random] context. 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/host-allocator-random/spring-host-allocator-random-context.xml] 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:28,591 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-allocator-random] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:28,686 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.agent.manager.allocator.impl.RandomAllocator@4b68d2a6 2025-06-29 03:36:28,686 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:28,686 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RandomAllocator. 2025-06-29 03:36:28,686 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:28,686 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [host-allocator-random] in 95 ms 2025-06-29 03:36:28,686 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [planner] context. 2025-06-29 03:36:28,686 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [planner] context. 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:28,687 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:28,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:28,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [planner] in 80 ms 2025-06-29 03:36:28,767 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [api-planner] context. 2025-06-29 03:36:28,767 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [api-planner] context. 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api-planner/spring-api-planner-context.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:28,767 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:28,857 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserDispersingPlanner 2025-06-29 03:36:28,897 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [UserDispersingPlanner] in [Deployment Planners Registry] 2025-06-29 03:36:28,897 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.deploy.UserDispersingPlanner@6c92af74 2025-06-29 03:36:28,897 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:28,897 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserDispersingPlanner. 2025-06-29 03:36:28,902 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:28,902 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [api-planner] in 135 ms 2025-06-29 03:36:28,902 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-planner] context. 2025-06-29 03:36:28,903 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [baremetal-planner] context. 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/baremetal-planner/spring-baremetal-planner-context.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:28,903 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:28,994 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BareMetalPlanner] in [Deployment Planners Registry] 2025-06-29 03:36:28,994 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.manager.BareMetalPlanner@14c16388 2025-06-29 03:36:28,994 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:28,994 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BareMetalPlanner. 2025-06-29 03:36:28,994 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [baremetal-planner] in 92 ms 2025-06-29 03:36:28,995 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [explicit-dedication] context. 2025-06-29 03:36:28,995 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [explicit-dedication] context. 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/explicit-dedication/spring-explicit-dedication-context.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:28,995 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [explicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,094 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ExplicitDedicationProcessor] in [Affinity Processors Registry] 2025-06-29 03:36:29,094 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.affinity.ExplicitDedicationProcessor@4a04177e 2025-06-29 03:36:29,094 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,094 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ExplicitDedicationProcessor. 2025-06-29 03:36:29,094 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,094 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [explicit-dedication] in 99 ms 2025-06-29 03:36:29,094 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [host-affinity] context. 2025-06-29 03:36:29,094 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [host-affinity] context. 2025-06-29 03:36:29,094 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/host-affinity/spring-host-affinity-context.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,186 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HostAffinityProcessor] in [Affinity Processors Registry] 2025-06-29 03:36:29,186 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.affinity.HostAffinityProcessor@5418225f 2025-06-29 03:36:29,186 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,186 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostAffinityProcessor. 2025-06-29 03:36:29,186 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [host-affinity] in 93 ms 2025-06-29 03:36:29,187 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [host-anti-affinity] context. 2025-06-29 03:36:29,187 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [host-anti-affinity] context. 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/host-anti-affinity/spring-host-anti-affinity-context.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,187 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,279 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HostAntiAffinityProcessor] in [Affinity Processors Registry] 2025-06-29 03:36:29,279 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.affinity.HostAntiAffinityProcessor@61736817 2025-06-29 03:36:29,280 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,280 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HostAntiAffinityProcessor. 2025-06-29 03:36:29,281 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,281 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [host-anti-affinity] in 94 ms 2025-06-29 03:36:29,282 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [implicit-dedication] context. 2025-06-29 03:36:29,282 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [implicit-dedication] context. 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/implicit-dedication/spring-implicit-dedication-context.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,282 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [implicit-dedication] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,366 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ImplicitDedicationPlanner 2025-06-29 03:36:29,441 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ImplicitDedicationPlanner] in [Deployment Planners Registry] 2025-06-29 03:36:29,442 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.deploy.ImplicitDedicationPlanner@349e6ca8 2025-06-29 03:36:29,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,442 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ImplicitDedicationPlanner. 2025-06-29 03:36:29,445 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,446 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [implicit-dedication] in 164 ms 2025-06-29 03:36:29,446 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [non-strict-host-affinity] context. 2025-06-29 03:36:29,446 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [non-strict-host-affinity] context. 2025-06-29 03:36:29,446 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/non-strict-host-affinity/spring-non-strict-host-affinity-context.xml] 2025-06-29 03:36:29,446 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,447 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,447 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,447 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,447 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,447 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,447 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,553 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NonStrictHostAffinityProcessor] in [Affinity Processors Registry] 2025-06-29 03:36:29,553 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.affinity.NonStrictHostAffinityProcessor@31eed3bc 2025-06-29 03:36:29,553 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,554 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NonStrictHostAffinityProcessor. 2025-06-29 03:36:29,555 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [non-strict-host-affinity] in 110 ms 2025-06-29 03:36:29,556 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [non-strict-host-anti-affinity] context. 2025-06-29 03:36:29,556 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [non-strict-host-anti-affinity] context. 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/non-strict-host-anti-affinity/spring-non-strict-host-anti-affinity-context.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,556 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [non-strict-host-anti-affinity] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,645 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NonStrictHostAntiAffinityProcessor] in [Affinity Processors Registry] 2025-06-29 03:36:29,645 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.affinity.NonStrictHostAntiAffinityProcessor@467c785d 2025-06-29 03:36:29,645 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,646 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NonStrictHostAntiAffinityProcessor. 2025-06-29 03:36:29,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,647 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [non-strict-host-anti-affinity] in 91 ms 2025-06-29 03:36:29,647 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-planner] context. 2025-06-29 03:36:29,647 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-planner] context. 2025-06-29 03:36:29,647 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-planner/spring-server-planner-context.xml] 2025-06-29 03:36:29,647 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,647 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,647 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,647 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-planner] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,731 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from FirstFitPlanner 2025-06-29 03:36:29,766 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [FirstFitPlanner] in [Deployment Planners Registry] 2025-06-29 03:36:29,766 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.deploy.FirstFitPlanner@3f64a088 2025-06-29 03:36:29,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirstFitPlanner. 2025-06-29 03:36:29,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-planner] in 122 ms 2025-06-29 03:36:29,769 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [skip-heurestics] context. 2025-06-29 03:36:29,769 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [skip-heurestics] context. 2025-06-29 03:36:29,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/skip-heurestics/spring-skip-heurestics-context.xml] 2025-06-29 03:36:29,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,770 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,770 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,770 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,770 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,770 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [skip-heurestics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,857 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SkipHeuresticsPlanner 2025-06-29 03:36:29,894 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.deploy.SkipHeuresticsPlanner@47f74d8f 2025-06-29 03:36:29,896 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SkipHeuresticsPlanner] in [Ha Planners Registry] 2025-06-29 03:36:29,897 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.deploy.SkipHeuresticsPlanner@47f74d8f 2025-06-29 03:36:29,897 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:29,897 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SkipHeuresticsPlanner. 2025-06-29 03:36:29,899 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [skip-heurestics] in 131 ms 2025-06-29 03:36:29,900 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [user-concentrated-pod] context. 2025-06-29 03:36:29,900 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [user-concentrated-pod] context. 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/user-concentrated-pod/spring-user-concentrated-pod-context.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/planner/spring-core-lifecycle-planner-context-inheritable.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:29,900 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [user-concentrated-pod] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:29,980 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserConcentratedPodPlanner 2025-06-29 03:36:30,028 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [UserConcentratedPodPlanner] in [Deployment Planners Registry] 2025-06-29 03:36:30,028 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.deploy.UserConcentratedPodPlanner@55cb0354 2025-06-29 03:36:30,028 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,028 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserConcentratedPodPlanner. 2025-06-29 03:36:30,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,031 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [user-concentrated-pod] in 131 ms 2025-06-29 03:36:30,032 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-allocator] context. 2025-06-29 03:36:30,032 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-allocator] context. 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-allocator/spring-server-allocator-context.xml] 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/allocator/spring-core-lifecycle-allocator-context-inheritable.xml] 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,124 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.agent.manager.allocator.impl.RecreateHostAllocator@6362dcf2 2025-06-29 03:36:30,124 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.agent.manager.allocator.impl.FirstFitAllocator@3045fbb4 2025-06-29 03:36:30,125 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [FirstFitRouting] in [Host Allocators Registry] 2025-06-29 03:36:30,125 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.agent.manager.allocator.impl.FirstFitRoutingAllocator@52602c48 2025-06-29 03:36:30,125 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.agent.manager.allocator.impl.TestingAllocator@4febb6e6 2025-06-29 03:36:30,127 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Balance] in [Console Proxy Allocators Registry] 2025-06-29 03:36:30,127 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.consoleproxy.ConsoleProxyBalanceAllocator@77f0b9d7 2025-06-29 03:36:30,127 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,127 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RecreateHostAllocator. 2025-06-29 03:36:30,127 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirstFitAllocator. 2025-06-29 03:36:30,127 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Balance. 2025-06-29 03:36:30,127 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean FirstFitRouting. 2025-06-29 03:36:30,127 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TestingAllocator. 2025-06-29 03:36:30,128 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-allocator] in 96 ms 2025-06-29 03:36:30,128 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [api] context. 2025-06-29 03:36:30,128 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [api] context. 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,200 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,200 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [api] in 72 ms 2025-06-29 03:36:30,200 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [acl-dynamic-role-based] context. 2025-06-29 03:36:30,200 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [acl-dynamic-role-based] context. 2025-06-29 03:36:30,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/acl-dynamic-role-based/spring-acl-dynamic-role-based-context.xml] 2025-06-29 03:36:30,200 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-dynamic-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,313 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DynamicRoleBasedAPIAccessChecker] in [Api Checkers Registry] 2025-06-29 03:36:30,313 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.DynamicRoleBasedAPIAccessChecker@31e5f3f0 2025-06-29 03:36:30,315 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DynamicRoleBasedAPIAccessChecker] in [Api Acl Checkers Registry] 2025-06-29 03:36:30,315 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.DynamicRoleBasedAPIAccessChecker@31e5f3f0 2025-06-29 03:36:30,315 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,315 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DynamicRoleBasedAPIAccessChecker. 2025-06-29 03:36:30,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [acl-dynamic-role-based] in 119 ms 2025-06-29 03:36:30,319 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [acl-project-role-based] context. 2025-06-29 03:36:30,319 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [acl-project-role-based] context. 2025-06-29 03:36:30,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/acl-project-role-based/spring-acl-project-role-based-context.xml] 2025-06-29 03:36:30,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-project-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,405 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ProjectRoleBasedApiAccessChecker] in [Api Checkers Registry] 2025-06-29 03:36:30,405 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.ProjectRoleBasedApiAccessChecker@66ed4881 2025-06-29 03:36:30,406 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ProjectRoleBasedApiAccessChecker] in [Api Acl Checkers Registry] 2025-06-29 03:36:30,406 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.ProjectRoleBasedApiAccessChecker@66ed4881 2025-06-29 03:36:30,406 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,406 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ProjectRoleBasedApiAccessChecker. 2025-06-29 03:36:30,406 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [acl-project-role-based] in 88 ms 2025-06-29 03:36:30,407 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [acl-static-role-based] context. 2025-06-29 03:36:30,407 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [acl-static-role-based] context. 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/acl-static-role-based/spring-acl-static-role-based-context.xml] 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,407 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [acl-static-role-based] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,495 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StaticRoleBasedAPIAccessChecker] in [Api Checkers Registry] 2025-06-29 03:36:30,495 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.StaticRoleBasedAPIAccessChecker@6f2c2278 2025-06-29 03:36:30,497 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StaticRoleBasedAPIAccessChecker] in [Api Acl Checkers Registry] 2025-06-29 03:36:30,497 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.acl.StaticRoleBasedAPIAccessChecker@6f2c2278 2025-06-29 03:36:30,497 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,497 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StaticRoleBasedAPIAccessChecker. 2025-06-29 03:36:30,499 ERROR [c.c.u.PropertiesUtil] (main:[]) (logid:) Unable to find properties file: commands.properties 2025-06-29 03:36:30,499 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,499 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [acl-static-role-based] in 92 ms 2025-06-29 03:36:30,499 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [cloudian] context. 2025-06-29 03:36:30,500 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [cloudian] context. 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/cloudian/spring-cloudian-context.xml] 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,500 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,584 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from CloudianConnectorImpl 2025-06-29 03:36:30,616 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CloudianConnectorImpl] in [Api Commands Registry] 2025-06-29 03:36:30,616 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.cloudian.CloudianConnectorImpl@2de08488 2025-06-29 03:36:30,618 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,618 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CloudianConnectorImpl. 2025-06-29 03:36:30,618 DEBUG [o.a.c.c.CloudianConnectorImpl] (main:[]) (logid:) Cloudian connector is disabled, skipping configuration 2025-06-29 03:36:30,619 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [cloudian] in 119 ms 2025-06-29 03:36:30,619 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ldap] context. 2025-06-29 03:36:30,619 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ldap] context. 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ldap/spring-ldap-context.xml] 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,619 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ldap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,728 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from LdapConfiguration 2025-06-29 03:36:30,832 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [LDAP] in [User Authenticators Registry] 2025-06-29 03:36:30,832 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ldap.LdapAuthenticator@2bab0d9a 2025-06-29 03:36:30,835 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for org.apache.cloudstack.ldap.LdapAuthenticator@2bab0d9a 2025-06-29 03:36:30,835 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [LdapManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:30,835 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ldap.LdapManagerImpl@21618f32 2025-06-29 03:36:30,839 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,839 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LdapConfigurationDaoImpl. 2025-06-29 03:36:30,840 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LdapTrustMapDaoImpl. 2025-06-29 03:36:30,840 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LDAP. 2025-06-29 03:36:30,840 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LdapManagerImpl. 2025-06-29 03:36:30,840 DEBUG [o.a.c.l.LdapManagerImpl] (main:[]) (logid:) Configuring LDAP Manager 2025-06-29 03:36:30,841 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ldap] in 222 ms 2025-06-29 03:36:30,841 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [maintenance] context. 2025-06-29 03:36:30,841 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [maintenance] context. 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/maintenance/spring-maintenance-context.xml] 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,841 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [maintenance] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:30,913 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:30,930 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [managementServerMaintenanceManager] in [Api Commands Registry] 2025-06-29 03:36:30,930 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.maintenance.ManagementServerMaintenanceManagerImpl@38bbbb3c 2025-06-29 03:36:30,930 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:30,930 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean managementServerMaintenanceManager. 2025-06-29 03:36:30,930 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [maintenance] in 90 ms 2025-06-29 03:36:30,931 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [md5] context. 2025-06-29 03:36:30,931 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [md5] context. 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/md5/spring-md5-context.xml] 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:30,931 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [md5] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:31,003 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [MD5] in [User Authenticators Registry] 2025-06-29 03:36:31,003 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.auth.MD5UserAuthenticator@70fbab13 2025-06-29 03:36:31,004 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for org.apache.cloudstack.auth.MD5UserAuthenticator@70fbab13 2025-06-29 03:36:31,004 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:31,004 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean MD5. 2025-06-29 03:36:31,004 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:31,004 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [md5] in 73 ms 2025-06-29 03:36:31,004 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [metrics] context. 2025-06-29 03:36:31,005 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [metrics] context. 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/metrics/spring-metrics-context.xml] 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:31,005 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [metrics] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:31,083 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from MetricsServiceImpl 2025-06-29 03:36:31,100 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [MetricsServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:31,100 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.metrics.MetricsServiceImpl@50f71f48 2025-06-29 03:36:31,103 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:31,103 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean MetricsServiceImpl. 2025-06-29 03:36:31,103 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:31,103 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [metrics] in 98 ms 2025-06-29 03:36:31,103 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [oauth2] context. 2025-06-29 03:36:31,103 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [oauth2] context. 2025-06-29 03:36:31,103 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/oauth2/spring-oauth2-context.xml] 2025-06-29 03:36:31,103 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:31,103 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:31,103 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:31,104 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:31,104 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:31,104 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [oauth2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:31,202 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from OAuth2AuthManagerImpl 2025-06-29 03:36:31,236 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [github] in [User O Auth2 Authenticators Registry] 2025-06-29 03:36:31,236 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.oauth2.github.GithubOAuth2Provider@65af4162 2025-06-29 03:36:31,236 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [google] in [User O Auth2 Authenticators Registry] 2025-06-29 03:36:31,236 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.oauth2.google.GoogleOAuth2Provider@418287ac 2025-06-29 03:36:31,236 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [oauth2] in [User Authenticators Registry] 2025-06-29 03:36:31,236 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.oauth2.OAuth2UserAuthenticator@d74f68e 2025-06-29 03:36:31,240 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OAUTH2Auth] in [Pluggable A P I Authenticators Registry] 2025-06-29 03:36:31,240 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.oauth2.OAuth2AuthManagerImpl@389ae8f1 2025-06-29 03:36:31,240 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [oauth2] in [User Password Encoders Registry] 2025-06-29 03:36:31,241 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.oauth2.OAuth2UserAuthenticator@d74f68e 2025-06-29 03:36:31,241 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OAUTH2Auth] in [Api Commands Registry] 2025-06-29 03:36:31,241 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.oauth2.OAuth2AuthManagerImpl@389ae8f1 2025-06-29 03:36:31,241 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:31,241 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OauthProviderDaoImpl. 2025-06-29 03:36:31,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OAUTH2Auth. 2025-06-29 03:36:31,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean oauth2. 2025-06-29 03:36:31,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean github. 2025-06-29 03:36:31,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean google. 2025-06-29 03:36:31,242 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:31,242 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [oauth2] in 139 ms 2025-06-29 03:36:31,242 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [pbkdf2] context. 2025-06-29 03:36:31,242 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [pbkdf2] context. 2025-06-29 03:36:31,242 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/pbkdf2/spring-pbkdf2-context.xml] 2025-06-29 03:36:31,242 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:31,242 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:31,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:31,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:31,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:31,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [pbkdf2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:31,317 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PBKDF2] in [User Authenticators Registry] 2025-06-29 03:36:31,318 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.server.auth.PBKDF2UserAuthenticator@5ad59bed 2025-06-29 03:36:31,318 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PBKDF2] in [User Password Encoders Registry] 2025-06-29 03:36:31,318 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.server.auth.PBKDF2UserAuthenticator@5ad59bed 2025-06-29 03:36:31,318 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:31,318 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PBKDF2. 2025-06-29 03:36:31,318 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:31,318 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [pbkdf2] in 76 ms 2025-06-29 03:36:31,318 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [plaintext] context. 2025-06-29 03:36:31,318 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [plaintext] context. 2025-06-29 03:36:31,318 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/plaintext/spring-plaintext-context.xml] 2025-06-29 03:36:31,318 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:31,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:31,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:31,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:31,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:31,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [plaintext] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:31,409 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for org.apache.cloudstack.auth.PlainTextUserAuthenticator@4452b545 2025-06-29 03:36:31,409 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for org.apache.cloudstack.auth.PlainTextUserAuthenticator@4452b545 2025-06-29 03:36:31,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:31,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PLAINTEXT. 2025-06-29 03:36:31,409 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [plaintext] in 92 ms 2025-06-29 03:36:31,410 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [prometheus] context. 2025-06-29 03:36:31,410 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [prometheus] context. 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/prometheus/spring-prometheus-context.xml] 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:31,410 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [prometheus] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:31,494 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from PrometheusExporterServerImpl 2025-06-29 03:36:31,543 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:31,543 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PrometheusExporterImpl. 2025-06-29 03:36:31,543 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PrometheusExporterServerImpl. 2025-06-29 03:36:31,543 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:31,543 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [prometheus] in 133 ms 2025-06-29 03:36:31,543 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [quota] context. 2025-06-29 03:36:31,543 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [quota] context. 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/quota/spring-framework-quota-context.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/quota/spring-quota-context.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:31,544 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [quota] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,091 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from QuotaServiceImpl 2025-06-29 03:36:32,140 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [QuotaServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:32,140 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.quota.QuotaServiceImpl@52b3a9c6 2025-06-29 03:36:32,143 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,143 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaTariffDaoImpl. 2025-06-29 03:36:32,143 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaEmailConfigurationDaoImpl. 2025-06-29 03:36:32,143 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaAccountDaoImpl. 2025-06-29 03:36:32,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaBalanceDaoImpl. 2025-06-29 03:36:32,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaCreditsDaoImpl. 2025-06-29 03:36:32,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaEmailTemplatesDaoImpl. 2025-06-29 03:36:32,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaUsageDaoImpl. 2025-06-29 03:36:32,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserVmDetailsDaoImpl. 2025-06-29 03:36:32,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaServiceImpl. 2025-06-29 03:36:32,146 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaAlertManagerImpl. 2025-06-29 03:36:32,146 DEBUG [o.a.c.u.m.SMTPMailSender] (main:[]) (logid:) Unable to instantiate SMTP mail session due to empty or null host. 2025-06-29 03:36:32,146 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaStatementImpl. 2025-06-29 03:36:32,147 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean QuotaManagerImpl. 2025-06-29 03:36:32,147 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [quota] in 603 ms 2025-06-29 03:36:32,147 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [rate-limit] context. 2025-06-29 03:36:32,147 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [rate-limit] context. 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/rate-limit/spring-rate-limit-context.xml] 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,147 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,148 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [rate-limit] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,237 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ApiRateLimitServiceImpl] in [Api Checkers Registry] 2025-06-29 03:36:32,237 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ratelimit.ApiRateLimitServiceImpl@4d4c0408 2025-06-29 03:36:32,237 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ApiRateLimitServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:32,237 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ratelimit.ApiRateLimitServiceImpl@4d4c0408 2025-06-29 03:36:32,238 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,238 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApiRateLimitServiceImpl. 2025-06-29 03:36:32,356 INFO [o.a.c.r.ApiRateLimitServiceImpl] (main:[]) (logid:) Limit Cache created with timeToLive=1, maxAllowed=25, maxElements=50000 2025-06-29 03:36:32,360 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [rate-limit] in 214 ms 2025-06-29 03:36:32,361 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [saml2] context. 2025-06-29 03:36:32,361 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [saml2] context. 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/saml2/spring-saml2-context.xml] 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [saml2] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,516 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SAML2AuthManagerImpl 2025-06-29 03:36:32,621 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SAML2] in [User Authenticators Registry] 2025-06-29 03:36:32,621 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.saml.SAML2UserAuthenticator@5c7712f0 2025-06-29 03:36:32,623 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SAML2Auth] in [Pluggable A P I Authenticators Registry] 2025-06-29 03:36:32,623 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.saml.SAML2AuthManagerImpl@12bd2630 2025-06-29 03:36:32,623 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SAML2] in [User Password Encoders Registry] 2025-06-29 03:36:32,623 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.saml.SAML2UserAuthenticator@5c7712f0 2025-06-29 03:36:32,623 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SAML2Auth] in [Api Commands Registry] 2025-06-29 03:36:32,623 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.saml.SAML2AuthManagerImpl@12bd2630 2025-06-29 03:36:32,624 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,624 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SAMLTokenDaoImpl. 2025-06-29 03:36:32,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SAML2. 2025-06-29 03:36:32,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SAML2Auth. 2025-06-29 03:36:32,625 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,625 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [saml2] in 264 ms 2025-06-29 03:36:32,625 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-api] context. 2025-06-29 03:36:32,625 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-api] context. 2025-06-29 03:36:32,625 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-api/spring-server-api-context.xml] 2025-06-29 03:36:32,625 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,625 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,625 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,625 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,626 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,626 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-api] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,702 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DomainChecker] in [Security Checkers Registry] 2025-06-29 03:36:32,702 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.acl.DomainChecker@2c15515e 2025-06-29 03:36:32,703 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [AffinityGroupAccessChecker] in [Security Checkers Registry] 2025-06-29 03:36:32,703 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.acl.AffinityGroupAccessChecker@50a65fb9 2025-06-29 03:36:32,703 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,703 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DomainChecker. 2025-06-29 03:36:32,703 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean AffinityGroupAccessChecker. 2025-06-29 03:36:32,703 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-api] in 79 ms 2025-06-29 03:36:32,704 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [sha256salted] context. 2025-06-29 03:36:32,704 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [sha256salted] context. 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/sha256salted/spring-sha256salted-context.xml] 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sha256salted] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,768 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SHA256SALT] in [User Authenticators Registry] 2025-06-29 03:36:32,768 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.auth.SHA256SaltedUserAuthenticator@1fbe8592 2025-06-29 03:36:32,768 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SHA256SALT] in [User Password Encoders Registry] 2025-06-29 03:36:32,768 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.auth.SHA256SaltedUserAuthenticator@1fbe8592 2025-06-29 03:36:32,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SHA256SALT. 2025-06-29 03:36:32,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [sha256salted] in 65 ms 2025-06-29 03:36:32,769 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [solidfire-intg-test] context. 2025-06-29 03:36:32,769 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [solidfire-intg-test] context. 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/solidfire-intg-test/spring-solidfire-intg-test-context.xml] 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,769 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [solidfire-intg-test] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,840 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ApiSolidFireIntegrationTestServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:32,840 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.api.solidfire.ApiSolidFireIntegrationTestServiceImpl@2dc33a2d 2025-06-29 03:36:32,841 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,841 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApiSolidFireIntegrationTestServiceImpl. 2025-06-29 03:36:32,841 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [solidfire-intg-test] in 73 ms 2025-06-29 03:36:32,842 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [staticpin] context. 2025-06-29 03:36:32,842 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [staticpin] context. 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/staticpin/spring-staticpin-context.xml] 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,842 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [staticpin] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,913 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [staticpin] in [User Two Factor Authenticators Registry] 2025-06-29 03:36:32,913 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.auth.StaticPinUserTwoFactorAuthenticator@598dca0a 2025-06-29 03:36:32,913 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,913 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean staticpin. 2025-06-29 03:36:32,913 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,913 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [staticpin] in 71 ms 2025-06-29 03:36:32,913 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [totp] context. 2025-06-29 03:36:32,913 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [totp] context. 2025-06-29 03:36:32,913 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/totp/spring-google-context.xml] 2025-06-29 03:36:32,913 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,914 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,914 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,914 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,914 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,914 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [totp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:32,980 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [totp] in [User Two Factor Authenticators Registry] 2025-06-29 03:36:32,980 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.auth.TotpUserTwoFactorAuthenticator@507eda0c 2025-06-29 03:36:32,980 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:32,980 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean totp. 2025-06-29 03:36:32,980 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:32,980 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [totp] in 67 ms 2025-06-29 03:36:32,980 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vmware-sioc] context. 2025-06-29 03:36:32,980 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vmware-sioc] context. 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vmware-sioc/spring-sioc-context.xml] 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api/spring-core-lifecycle-api-context-inheritable.xml] 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:32,981 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-sioc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,058 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ApiSiocServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:33,058 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.api.sioc.ApiSiocServiceImpl@4c5bcc6e 2025-06-29 03:36:33,058 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,058 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ApiSiocServiceImpl. 2025-06-29 03:36:33,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vmware-sioc] in 78 ms 2025-06-29 03:36:33,059 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [api-config] context. 2025-06-29 03:36:33,059 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [api-config] context. 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-config] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/api-config/spring-api-config-context.xml] 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-config] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-config] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-config] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-config] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [api-config] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,108 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ApiServiceConfiguration 2025-06-29 03:36:33,136 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,136 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,136 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [api-config] in 77 ms 2025-06-29 03:36:33,136 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [backend] context. 2025-06-29 03:36:33,136 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [backend] context. 2025-06-29 03:36:33,137 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,137 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,137 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,137 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,137 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,185 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,185 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,185 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [backend] in 48 ms 2025-06-29 03:36:33,185 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [backup] context. 2025-06-29 03:36:33,186 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [backup] context. 2025-06-29 03:36:33,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/backup/spring-core-lifecycle-backup-context-inheritable.xml] 2025-06-29 03:36:33,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,243 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,243 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [backup] in 57 ms 2025-06-29 03:36:33,243 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [dummy-backup] context. 2025-06-29 03:36:33,243 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [dummy-backup] context. 2025-06-29 03:36:33,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/dummy-backup/spring-backup-dummy-context.xml] 2025-06-29 03:36:33,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/backup/spring-core-lifecycle-backup-context-inheritable.xml] 2025-06-29 03:36:33,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,243 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,244 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,244 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [dummy-backup] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,312 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [dummy] in [Backup Providers Registry] 2025-06-29 03:36:33,312 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.backup.DummyBackupProvider@279ebd4 2025-06-29 03:36:33,312 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,312 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean dummy. 2025-06-29 03:36:33,312 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,313 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [dummy-backup] in 70 ms 2025-06-29 03:36:33,313 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [nas] context. 2025-06-29 03:36:33,313 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [nas] context. 2025-06-29 03:36:33,313 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/nas/spring-backup-nas-context.xml] 2025-06-29 03:36:33,313 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/backup/spring-core-lifecycle-backup-context-inheritable.xml] 2025-06-29 03:36:33,313 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,313 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,313 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,314 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,314 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nas] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,429 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NASBackupProvider 2025-06-29 03:36:33,433 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [nas] in [Backup Providers Registry] 2025-06-29 03:36:33,433 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.backup.NASBackupProvider@61938fae 2025-06-29 03:36:33,433 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,433 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean nas. 2025-06-29 03:36:33,434 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [nas] in 121 ms 2025-06-29 03:36:33,434 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [networker] context. 2025-06-29 03:36:33,434 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [networker] context. 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/networker/spring-backup-networker-context.xml] 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/backup/spring-core-lifecycle-backup-context-inheritable.xml] 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,434 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [networker] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,532 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NetworkerBackupProvider 2025-06-29 03:36:33,559 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [networker] in [Backup Providers Registry] 2025-06-29 03:36:33,559 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.backup.NetworkerBackupProvider@261a0d86 2025-06-29 03:36:33,559 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,559 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean networker. 2025-06-29 03:36:33,559 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [networker] in 126 ms 2025-06-29 03:36:33,560 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [veeam] context. 2025-06-29 03:36:33,560 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [veeam] context. 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/veeam/spring-backup-veeam-context.xml] 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/backup/spring-core-lifecycle-backup-context-inheritable.xml] 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,560 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [veeam] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,622 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VeeamBackupProvider 2025-06-29 03:36:33,651 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [veeam] in [Backup Providers Registry] 2025-06-29 03:36:33,651 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.backup.VeeamBackupProvider@21c09e90 2025-06-29 03:36:33,651 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,652 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean veeam. 2025-06-29 03:36:33,652 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [veeam] in 92 ms 2025-06-29 03:36:33,652 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ca] context. 2025-06-29 03:36:33,652 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ca] context. 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ca/spring-core-lifecycle-ca-context-inheritable.xml] 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,652 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,703 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,703 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:33,703 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ca] in 51 ms 2025-06-29 03:36:33,703 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [root-ca] context. 2025-06-29 03:36:33,703 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [root-ca] context. 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/root-ca/spring-root-ca-context.xml] 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ca/spring-core-lifecycle-ca-context-inheritable.xml] 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:33,704 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:33,766 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RootCAProvider 2025-06-29 03:36:33,790 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [RootCAProvider] in [Ca Providers Registry] 2025-06-29 03:36:33,790 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ca.provider.RootCAProvider@2c9d3df 2025-06-29 03:36:33,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:33,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider. 2025-06-29 03:36:33,947 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ]. 2025-06-29 03:36:33,957 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [4064598] for command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ]. 2025-06-29 03:36:33,960 DEBUG [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Trying to find management IP in CIDR range [172.16.20.0/24]. 2025-06-29 03:36:33,962 DEBUG [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Added IP [172.16.20.40] to the list of IPs in the management server's certificate. 2025-06-29 03:36:34,971 INFO [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Creating new management server certificate and keystore 2025-06-29 03:36:34,977 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:34,977 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [root-ca] in 1273 ms 2025-06-29 03:36:34,977 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [compute] context. 2025-06-29 03:36:34,977 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [compute] context. 2025-06-29 03:36:34,978 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:34,978 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:34,978 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:34,978 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:34,978 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:34,978 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [compute] in 58 ms 2025-06-29 03:36:35,036 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-compute] context. 2025-06-29 03:36:35,036 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [baremetal-compute] context. 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/baremetal-compute/spring-baremetal-compute-context.xml] 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,104 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BareMetalGuru 2025-06-29 03:36:35,127 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BaremetalGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:35,127 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.manager.BareMetalGuru@26300f29 2025-06-29 03:36:35,128 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,128 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BaremetalGuru. 2025-06-29 03:36:35,128 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [baremetal-compute] in 92 ms 2025-06-29 03:36:35,128 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [hyperv-compute] context. 2025-06-29 03:36:35,128 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [hyperv-compute] context. 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/hyperv-compute/spring-hyperv-compute-context.xml] 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,128 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,129 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,194 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HypervGuru 2025-06-29 03:36:35,222 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HypervInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:35,222 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.HypervInvestigator@7dcc5e9d 2025-06-29 03:36:35,223 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HypervGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:35,223 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.hyperv.guru.HypervGuru@47effce0 2025-06-29 03:36:35,223 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,223 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HypervInvestigator. 2025-06-29 03:36:35,223 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HypervGuru. 2025-06-29 03:36:35,223 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [hyperv-compute] in 96 ms 2025-06-29 03:36:35,224 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [kubernetes] context. 2025-06-29 03:36:35,224 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [kubernetes] context. 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/kubernetes/spring-core-lifecycle-kubernetes-context-inheritable.xml] 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,224 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,284 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,284 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,284 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [kubernetes] in 60 ms 2025-06-29 03:36:35,284 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [kubernetes-service] context. 2025-06-29 03:36:35,284 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [kubernetes-service] context. 2025-06-29 03:36:35,284 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/kubernetes-service/spring-kubernetes-service-context.xml] 2025-06-29 03:36:35,284 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/kubernetes/spring-core-lifecycle-kubernetes-context-inheritable.xml] 2025-06-29 03:36:35,284 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,284 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,284 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kubernetes-service] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,488 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KubernetesClusterManagerImpl 2025-06-29 03:36:35,597 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KubernetesServiceHelperImpl 2025-06-29 03:36:35,603 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KubernetesServiceHelper] in [Kubernetes Service Helper Registry] 2025-06-29 03:36:35,603 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.kubernetes.cluster.KubernetesServiceHelperImpl@261be6cf 2025-06-29 03:36:35,603 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KubernetesClusterManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:35,603 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl@7d83321a 2025-06-29 03:36:35,603 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KubernetesVersionManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:35,603 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.kubernetes.version.KubernetesVersionManagerImpl@16641839 2025-06-29 03:36:35,606 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,606 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesSupportedVersionDaoImpl. 2025-06-29 03:36:35,606 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesClusterDaoImpl. 2025-06-29 03:36:35,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesClusterVmMapDaoImpl. 2025-06-29 03:36:35,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesClusterDetailsDaoImpl. 2025-06-29 03:36:35,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesVersionManagerImpl. 2025-06-29 03:36:35,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesClusterManagerImpl. 2025-06-29 03:36:35,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KubernetesServiceHelper. 2025-06-29 03:36:35,607 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [kubernetes-service] in 324 ms 2025-06-29 03:36:35,608 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [kvm-compute] context. 2025-06-29 03:36:35,608 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [kvm-compute] context. 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/kvm-compute/spring-kvm-compute-context.xml] 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,608 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [kvm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,679 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KVMHAProvider 2025-06-29 03:36:35,723 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KVMInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:35,723 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.KVMInvestigator@348f470f 2025-06-29 03:36:35,723 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KVMHAProvider] in [Ha Providers Registry] 2025-06-29 03:36:35,723 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.kvm.ha.KVMHAProvider@4f3db580 2025-06-29 03:36:35,723 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,723 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KVMInvestigator. 2025-06-29 03:36:35,723 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KVMHostActivityChecker. 2025-06-29 03:36:35,723 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KVMHAProvider. 2025-06-29 03:36:35,723 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,723 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [kvm-compute] in 115 ms 2025-06-29 03:36:35,723 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ovm-compute] context. 2025-06-29 03:36:35,723 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ovm-compute] context. 2025-06-29 03:36:35,723 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ovm-compute/spring-ovm-compute-context.xml] 2025-06-29 03:36:35,723 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,724 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,724 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,724 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,724 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,724 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,785 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from OvmGuru 2025-06-29 03:36:35,807 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OvmFenceBuilder] in [Ha Fence Builders Registry] 2025-06-29 03:36:35,808 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ovm.hypervisor.OvmFencer@6983436f 2025-06-29 03:36:35,809 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OvmGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:35,809 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ovm.hypervisor.OvmGuru@5476f67a 2025-06-29 03:36:35,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvmFenceBuilder. 2025-06-29 03:36:35,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvmGuru. 2025-06-29 03:36:35,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ovm-compute] in 87 ms 2025-06-29 03:36:35,810 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ovm3-compute] context. 2025-06-29 03:36:35,810 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ovm3-compute] context. 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ovm3-compute/spring-ovm3-compute-context.xml] 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,869 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from Ovm3HypervisorGuru 2025-06-29 03:36:35,893 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovm3Investigator] in [Ha Investigators Registry] 2025-06-29 03:36:35,893 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.Ovm3Investigator@64073f50 2025-06-29 03:36:35,894 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovm3FenceBuilder] in [Ha Fence Builders Registry] 2025-06-29 03:36:35,894 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.ovm3.resources.Ovm3FenceBuilder@5a056d39 2025-06-29 03:36:35,895 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovm3Guru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:35,895 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.ovm3.resources.Ovm3HypervisorGuru@f90ff5e 2025-06-29 03:36:35,895 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,895 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ovm3Investigator. 2025-06-29 03:36:35,895 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ovm3FenceBuilder. 2025-06-29 03:36:35,895 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ovm3Guru. 2025-06-29 03:36:35,895 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,895 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ovm3-compute] in 85 ms 2025-06-29 03:36:35,895 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-alert-adapter-compute] context. 2025-06-29 03:36:35,895 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-alert-adapter-compute] context. 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-alert-adapter-compute/spring-server-alert-adapter-compute-context.xml] 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,896 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:35,958 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:35,958 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConsoleProxyAlertAdapter. 2025-06-29 03:36:35,958 INFO [c.c.a.ConsoleProxyAlertAdapter] (main:[]) (logid:) Start configuring console proxy alert manager : ConsoleProxyAlertAdapter 2025-06-29 03:36:35,959 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:35,959 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-alert-adapter-compute] in 63 ms 2025-06-29 03:36:35,960 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-compute] context. 2025-06-29 03:36:35,960 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-compute] context. 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-compute/spring-server-compute-context.xml] 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:35,960 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,037 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KVMGuru 2025-06-29 03:36:36,060 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from LXCGuru 2025-06-29 03:36:36,091 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UnmanagedVMsManagerImpl 2025-06-29 03:36:36,141 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KVMGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:36,141 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.KVMGuru@498ae73c 2025-06-29 03:36:36,143 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [LXCGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:36,143 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.LXCGuru@5747a07c 2025-06-29 03:36:36,143 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [UnmanagedVMsManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:36,143 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.vm.UnmanagedVMsManagerImpl@104d07a8 2025-06-29 03:36:36,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KVMGuru. 2025-06-29 03:36:36,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LXCGuru. 2025-06-29 03:36:36,144 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,144 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-compute] in 184 ms 2025-06-29 03:36:36,144 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-fencer] context. 2025-06-29 03:36:36,144 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-fencer] context. 2025-06-29 03:36:36,144 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-fencer/spring-server-fencer-context.xml] 2025-06-29 03:36:36,144 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:36,144 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,145 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,145 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,145 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,145 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-fencer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,213 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.ha.RecreatableFencer@21cefc6d 2025-06-29 03:36:36,213 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KVMFenceBuilder] in [Ha Fence Builders Registry] 2025-06-29 03:36:36,213 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.KVMFencer@6d77c929 2025-06-29 03:36:36,213 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,213 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RecreatableFencer. 2025-06-29 03:36:36,213 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KVMFenceBuilder. 2025-06-29 03:36:36,214 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-fencer] in 70 ms 2025-06-29 03:36:36,214 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-investigator] context. 2025-06-29 03:36:36,214 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-investigator] context. 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-investigator/spring-server-investigator-context.xml] 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,214 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-investigator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,313 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [XenServerInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:36,313 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.XenServerInvestigator@e75dc31 2025-06-29 03:36:36,315 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ManagementIPSysVMInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:36,315 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.ManagementIPSystemVMInvestigator@1552bdc6 2025-06-29 03:36:36,317 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SimpleInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:36,317 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.CheckOnAgentInvestigator@7f4bbb1c 2025-06-29 03:36:36,319 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PingInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:36,319 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.UserVmDomRInvestigator@5ccb2393 2025-06-29 03:36:36,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean XenServerInvestigator. 2025-06-29 03:36:36,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ManagementIPSysVMInvestigator. 2025-06-29 03:36:36,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SimpleInvestigator. 2025-06-29 03:36:36,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PingInvestigator. 2025-06-29 03:36:36,319 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,319 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-investigator] in 105 ms 2025-06-29 03:36:36,320 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vmware-compute] context. 2025-06-29 03:36:36,320 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vmware-compute] context. 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vmware-compute/spring-vmware-compute-context.xml] 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,320 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,415 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VMwareGuru 2025-06-29 03:36:36,446 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VMwareInvestigator] in [Ha Investigators Registry] 2025-06-29 03:36:36,447 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.VmwareInvestigator@7a364bfe 2025-06-29 03:36:36,447 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VMwareFenceBuilder] in [Ha Fence Builders Registry] 2025-06-29 03:36:36,447 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.VmwareFencer@204fb56d 2025-06-29 03:36:36,448 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VMwareGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:36,448 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.guru.VMwareGuru@bc60f71 2025-06-29 03:36:36,448 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMwareFenceBuilder. 2025-06-29 03:36:36,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMwareGuru. 2025-06-29 03:36:36,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMwareInvestigator. 2025-06-29 03:36:36,449 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vmware-compute] in 129 ms 2025-06-29 03:36:36,449 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [xenserver-compute] context. 2025-06-29 03:36:36,449 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [xenserver-compute] context. 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/xenserver-compute/spring-xenserver-compute-context.xml] 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/compute/spring-core-lifecycle-compute-context-inheritable.xml] 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,449 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-compute] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,522 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from XenServerGuru 2025-06-29 03:36:36,534 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [XenServerFenceBuilder] in [Ha Fence Builders Registry] 2025-06-29 03:36:36,534 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ha.XenServerFencer@1ec7388c 2025-06-29 03:36:36,536 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [XenServerGuru] in [Hypervisor Gurus Registry] 2025-06-29 03:36:36,537 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.XenServerGuru@6b3d3a9a 2025-06-29 03:36:36,537 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,537 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean XenServerGuru. 2025-06-29 03:36:36,537 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean XenServerFenceBuilder. 2025-06-29 03:36:36,537 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [xenserver-compute] in 88 ms 2025-06-29 03:36:36,537 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [network] context. 2025-06-29 03:36:36,537 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [network] context. 2025-06-29 03:36:36,538 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:36,538 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,538 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,538 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,538 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,538 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,595 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,595 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [network] in 58 ms 2025-06-29 03:36:36,596 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-network] context. 2025-06-29 03:36:36,596 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [baremetal-network] context. 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/baremetal-network/spring-baremetal-network-context.xml] 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,596 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,597 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,683 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BareMetalDhcp] in [Dhcp Providers Registry] 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.networkservice.BaremetalDhcpElement@be6b0d6 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BareMetalDhcp] in [Network Elements Registry] 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.networkservice.BaremetalDhcpElement@be6b0d6 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BareMetalUserdata] in [Network Elements Registry] 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.networkservice.BaremetalUserdataElement@6ea8e8bc 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BareMetalPxe] in [Network Elements Registry] 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.networkservice.BaremetalPxeElement@a57b4bd 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BaremetalDhcpManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:36,684 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.networkservice.BaremetalDhcpManagerImpl@7ec09f73 2025-06-29 03:36:36,684 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,684 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean null. 2025-06-29 03:36:36,684 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BareMetalDhcp. 2025-06-29 03:36:36,684 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BareMetalUserdata. 2025-06-29 03:36:36,684 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BareMetalPxe. 2025-06-29 03:36:36,684 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [baremetal-network] in 89 ms 2025-06-29 03:36:36,685 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [bigswitch] context. 2025-06-29 03:36:36,685 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [bigswitch] context. 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bigswitch/spring-bigswitch-context.xml] 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,685 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [bigswitch] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:36,794 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BigSwitchBcfGuestNetworkGuru 2025-06-29 03:36:36,848 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcf] in [Ip Deployers Registry] 2025-06-29 03:36:36,848 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BigSwitchBcfElement@4809035f 2025-06-29 03:36:36,850 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcfGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:36,850 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.BigSwitchBcfGuestNetworkGuru@6d194f34 2025-06-29 03:36:36,850 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcf] in [Network Elements Registry] 2025-06-29 03:36:36,850 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BigSwitchBcfElement@4809035f 2025-06-29 03:36:36,851 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcf] in [Firewall Service Providers Registry] 2025-06-29 03:36:36,851 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BigSwitchBcfElement@4809035f 2025-06-29 03:36:36,853 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcf] in [Static Nat Service Providers Registry] 2025-06-29 03:36:36,853 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BigSwitchBcfElement@4809035f 2025-06-29 03:36:36,854 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcf] in [Network A C L Service Providers Registry] 2025-06-29 03:36:36,854 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BigSwitchBcfElement@4809035f 2025-06-29 03:36:36,854 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BigSwitchBcf] in [Api Commands Registry] 2025-06-29 03:36:36,854 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BigSwitchBcfElement@4809035f 2025-06-29 03:36:36,854 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:36,855 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BigSwitchBcfDaoImpl. 2025-06-29 03:36:36,855 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BigSwitchBcf. 2025-06-29 03:36:36,855 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BigSwitchBcfGuestNetworkGuru. 2025-06-29 03:36:36,855 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:36,855 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [bigswitch] in 170 ms 2025-06-29 03:36:36,855 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [cisco-vnmc] context. 2025-06-29 03:36:36,855 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [cisco-vnmc] context. 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/cisco-vnmc/spring-cisco-vnmc-context.xml] 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:36,856 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cisco-vnmc] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,011 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoVNMC] in [Ip Deployers Registry] 2025-06-29 03:36:37,011 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoVnmcElement@4f0f76b4 2025-06-29 03:36:37,012 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoVNMC] in [Network Elements Registry] 2025-06-29 03:36:37,012 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoVnmcElement@4f0f76b4 2025-06-29 03:36:37,013 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoVNMC] in [Firewall Service Providers Registry] 2025-06-29 03:36:37,013 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoVnmcElement@4f0f76b4 2025-06-29 03:36:37,014 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoVNMC] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:37,014 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoVnmcElement@4f0f76b4 2025-06-29 03:36:37,016 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoVNMC] in [Static Nat Service Providers Registry] 2025-06-29 03:36:37,016 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoVnmcElement@4f0f76b4 2025-06-29 03:36:37,016 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoVNMC] in [Api Commands Registry] 2025-06-29 03:36:37,016 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoVnmcElement@4f0f76b4 2025-06-29 03:36:37,017 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,017 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CiscoVnmcDaoImpl. 2025-06-29 03:36:37,017 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetworkAsa1000vMapDaoImpl. 2025-06-29 03:36:37,017 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CiscoAsa1000vDaoImpl. 2025-06-29 03:36:37,018 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CiscoVNMC. 2025-06-29 03:36:37,018 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,018 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [cisco-vnmc] in 162 ms 2025-06-29 03:36:37,018 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [contrail] context. 2025-06-29 03:36:37,018 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [contrail] context. 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/contrail/spring-contrail-context.xml] 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,019 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [contrail] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,168 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Ip Deployers Registry] 2025-06-29 03:36:37,168 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,169 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailElement] in [Ip Deployers Registry] 2025-06-29 03:36:37,169 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailElementImpl@38133bca 2025-06-29 03:36:37,170 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Dhcp Providers Registry] 2025-06-29 03:36:37,171 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,172 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailElement] in [Dhcp Providers Registry] 2025-06-29 03:36:37,172 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailElementImpl@38133bca 2025-06-29 03:36:37,174 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailGuru] in [Network Gurus Registry] 2025-06-29 03:36:37,174 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailGuru@515a3e3a 2025-06-29 03:36:37,174 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Network Elements Registry] 2025-06-29 03:36:37,174 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,174 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailElement] in [Network Elements Registry] 2025-06-29 03:36:37,174 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailElementImpl@38133bca 2025-06-29 03:36:37,176 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Vpc Providers Registry] 2025-06-29 03:36:37,176 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,177 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Static Nat Service Providers Registry] 2025-06-29 03:36:37,177 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,179 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailElement] in [Static Nat Service Providers Registry] 2025-06-29 03:36:37,179 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailElementImpl@38133bca 2025-06-29 03:36:37,182 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Network A C L Service Providers Registry] 2025-06-29 03:36:37,182 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,182 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailVpcElement] in [Api Commands Registry] 2025-06-29 03:36:37,182 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl@489b0369 2025-06-29 03:36:37,182 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ContrailElement] in [Api Commands Registry] 2025-06-29 03:36:37,182 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.contrail.management.ContrailElementImpl@38133bca 2025-06-29 03:36:37,183 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,183 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ContrailManagerImpl. 2025-06-29 03:36:37,184 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ContrailGuru. 2025-06-29 03:36:37,184 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ContrailVpcElement. 2025-06-29 03:36:37,184 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ContrailElement. 2025-06-29 03:36:37,185 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,185 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [contrail] in 166 ms 2025-06-29 03:36:37,185 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [elb] context. 2025-06-29 03:36:37,185 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [elb] context. 2025-06-29 03:36:37,185 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/elb/spring-elb-context.xml] 2025-06-29 03:36:37,185 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,186 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [elb] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,323 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,323 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ElasticLbVmMapDaoImpl. 2025-06-29 03:36:37,323 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ElasticLoadBalancerManagerImpl. 2025-06-29 03:36:37,338 INFO [c.c.n.l.ElasticLoadBalancerManagerImpl] (main:[]) (logid:) Elastic Load balancer enabled: false 2025-06-29 03:36:37,339 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,339 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [elb] in 154 ms 2025-06-29 03:36:37,339 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [globodns] context. 2025-06-29 03:36:37,339 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [globodns] context. 2025-06-29 03:36:37,339 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/globodns/spring-globodns-context.xml] 2025-06-29 03:36:37,339 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,339 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,339 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,340 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,340 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,340 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [globodns] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,461 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from GloboDnsElement 2025-06-29 03:36:37,485 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [GloboDnsElement] in [Network Elements Registry] 2025-06-29 03:36:37,485 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.globo.globodns.cloudstack.element.GloboDnsElement@5257f99d 2025-06-29 03:36:37,485 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [GloboDnsElement] in [Api Commands Registry] 2025-06-29 03:36:37,485 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.globo.globodns.cloudstack.element.GloboDnsElement@5257f99d 2025-06-29 03:36:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GloboDnsElement. 2025-06-29 03:36:37,486 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,486 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [globodns] in 147 ms 2025-06-29 03:36:37,487 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [netscaler] context. 2025-06-29 03:36:37,487 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [netscaler] context. 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/netscaler/spring-netscaler-context.xml] 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,487 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [netscaler] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,659 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Netscaler] in [Ip Deployers Registry] 2025-06-29 03:36:37,660 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NetscalerElement@4a4525a1 2025-06-29 03:36:37,660 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Netscaler] in [Network Elements Registry] 2025-06-29 03:36:37,660 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NetscalerElement@4a4525a1 2025-06-29 03:36:37,662 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Netscaler] in [Static Nat Service Providers Registry] 2025-06-29 03:36:37,662 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NetscalerElement@4a4525a1 2025-06-29 03:36:37,663 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Netscaler] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:37,663 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NetscalerElement@4a4525a1 2025-06-29 03:36:37,664 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Netscaler] in [Gslb Service Providers Registry] 2025-06-29 03:36:37,664 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NetscalerElement@4a4525a1 2025-06-29 03:36:37,665 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Netscaler] in [Api Commands Registry] 2025-06-29 03:36:37,665 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NetscalerElement@4a4525a1 2025-06-29 03:36:37,666 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,666 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetScalerPodDaoImpl. 2025-06-29 03:36:37,666 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetScalerServicePackageDaoImpl. 2025-06-29 03:36:37,666 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetScalerControlCenterDaoImpl. 2025-06-29 03:36:37,666 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NetScalerVMManagerImpl. 2025-06-29 03:36:37,667 INFO [c.c.n.v.NetScalerVMManagerImpl] (main:[]) (logid:) NetScalerVMManagerImpl has been configured 2025-06-29 03:36:37,667 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Netscaler. 2025-06-29 03:36:37,668 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,668 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [netscaler] in 181 ms 2025-06-29 03:36:37,669 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [nsx] context. 2025-06-29 03:36:37,669 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [nsx] context. 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/nsx/spring-nsx-context.xml] 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,669 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nsx] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,764 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NsxGuestNetworkGuru 2025-06-29 03:36:37,793 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Ip Deployers Registry] 2025-06-29 03:36:37,794 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,795 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Dhcp Providers Registry] 2025-06-29 03:36:37,795 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,797 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NsxPublicNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:37,797 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxPublicNetworkGuru@6e145703 2025-06-29 03:36:37,798 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NsxGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:37,798 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxGuestNetworkGuru@6e2ff120 2025-06-29 03:36:37,798 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Network Elements Registry] 2025-06-29 03:36:37,798 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,800 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Vpc Providers Registry] 2025-06-29 03:36:37,800 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,801 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Firewall Service Providers Registry] 2025-06-29 03:36:37,801 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,802 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:37,802 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,803 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Static Nat Service Providers Registry] 2025-06-29 03:36:37,803 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,804 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Network A C L Service Providers Registry] 2025-06-29 03:36:37,804 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Internal Load Balancer Element Service Registry] 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Nsx] in [Api Commands Registry] 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxElement@7c2c3098 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NsxProviderServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:37,806 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.service.NsxProviderServiceImpl@56be1b09 2025-06-29 03:36:37,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Nsx. 2025-06-29 03:36:37,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener NsxElement with id 23 2025-06-29 03:36:37,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NsxPublicNetworkGuru. 2025-06-29 03:36:37,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NsxGuestNetworkGuru. 2025-06-29 03:36:37,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [nsx] in 139 ms 2025-06-29 03:36:37,808 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [nvp] context. 2025-06-29 03:36:37,808 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [nvp] context. 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/nvp/spring-nvp-context.xml] 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,808 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nvp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:37,918 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from NiciraNvpGuestNetworkGuru 2025-06-29 03:36:37,958 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NiciraNvp] in [Ip Deployers Registry] 2025-06-29 03:36:37,958 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NiciraNvpElement@3bbd0a50 2025-06-29 03:36:37,960 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NiciraNvpGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:37,960 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.NiciraNvpGuestNetworkGuru@712d42ed 2025-06-29 03:36:37,960 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NiciraNvp] in [Network Elements Registry] 2025-06-29 03:36:37,960 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NiciraNvpElement@3bbd0a50 2025-06-29 03:36:37,962 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NiciraNvp] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:37,962 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NiciraNvpElement@3bbd0a50 2025-06-29 03:36:37,963 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NiciraNvp] in [Static Nat Service Providers Registry] 2025-06-29 03:36:37,963 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NiciraNvpElement@3bbd0a50 2025-06-29 03:36:37,963 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NiciraNvp] in [Api Commands Registry] 2025-06-29 03:36:37,963 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.NiciraNvpElement@3bbd0a50 2025-06-29 03:36:37,963 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:37,963 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NiciraNvpDaoImpl. 2025-06-29 03:36:37,963 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NiciraNvpNicMappingDaoImpl. 2025-06-29 03:36:37,964 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NiciraNvpRouterMappingDaoImpl. 2025-06-29 03:36:37,964 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NiciraNvpGuestNetworkGuru. 2025-06-29 03:36:37,964 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NiciraNvp. 2025-06-29 03:36:37,964 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:37,964 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [nvp] in 156 ms 2025-06-29 03:36:37,964 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [opendaylight] context. 2025-06-29 03:36:37,964 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [opendaylight] context. 2025-06-29 03:36:37,964 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/opendaylight/spring-opendaylight-context.xml] 2025-06-29 03:36:37,965 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:37,965 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:37,965 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:37,965 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:37,965 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:37,965 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [opendaylight] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:38,061 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from OpendaylightGuestNetworkGuru 2025-06-29 03:36:38,094 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OpendaylightGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,094 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.opendaylight.OpendaylightGuestNetworkGuru@e9fcd1e 2025-06-29 03:36:38,094 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OpendaylightElement] in [Network Elements Registry] 2025-06-29 03:36:38,094 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.opendaylight.OpendaylightElement@3990f312 2025-06-29 03:36:38,094 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OpenDaylightControllerResourceManagerImpl] in [Api Commands Registry] 2025-06-29 03:36:38,094 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.opendaylight.agent.OpenDaylightControllerResourceManagerImpl@4d031f38 2025-06-29 03:36:38,094 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:38,094 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OpenDaylightControllerMappingDaoImpl. 2025-06-29 03:36:38,095 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OpendaylightGuestNetworkGuru. 2025-06-29 03:36:38,095 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OpendaylightElement. 2025-06-29 03:36:38,095 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:38,095 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [opendaylight] in 131 ms 2025-06-29 03:36:38,095 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ovs] context. 2025-06-29 03:36:38,095 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ovs] context. 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ovs/spring-ovs-context.xml] 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:38,096 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:38,254 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from OvsGuestNetworkGuru 2025-06-29 03:36:38,278 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovs] in [Ip Deployers Registry] 2025-06-29 03:36:38,278 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.OvsElement@52386242 2025-06-29 03:36:38,280 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [OvsGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,280 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.OvsGuestNetworkGuru@7ab9d59f 2025-06-29 03:36:38,280 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovs] in [Network Elements Registry] 2025-06-29 03:36:38,280 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.OvsElement@52386242 2025-06-29 03:36:38,281 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovs] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:38,282 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.OvsElement@52386242 2025-06-29 03:36:38,283 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovs] in [Static Nat Service Providers Registry] 2025-06-29 03:36:38,283 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.OvsElement@52386242 2025-06-29 03:36:38,285 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovs] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:38,285 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.OvsElement@52386242 2025-06-29 03:36:38,285 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovs] in [Api Commands Registry] 2025-06-29 03:36:38,285 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.OvsElement@52386242 2025-06-29 03:36:38,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:38,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvsTunnelNetworkDaoImpl. 2025-06-29 03:36:38,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VpcDistributedRouterSeqNoDaoImpl. 2025-06-29 03:36:38,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvsTunnelInterfaceDaoImpl. 2025-06-29 03:36:38,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvsNetworkTopologyGuruImpl. 2025-06-29 03:36:38,286 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvsTunnelManagerImpl. 2025-06-29 03:36:38,287 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean OvsGuestNetworkGuru. 2025-06-29 03:36:38,287 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ovs. 2025-06-29 03:36:38,287 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:38,287 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ovs] in 192 ms 2025-06-29 03:36:38,287 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [paloalto] context. 2025-06-29 03:36:38,287 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [paloalto] context. 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/paloalto/spring-paloalto-context.xml] 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:38,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [paloalto] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:38,381 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PaloAlto] in [Ip Deployers Registry] 2025-06-29 03:36:38,381 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.PaloAltoExternalFirewallElement@456bff44 2025-06-29 03:36:38,381 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PaloAlto] in [Network Elements Registry] 2025-06-29 03:36:38,381 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.PaloAltoExternalFirewallElement@456bff44 2025-06-29 03:36:38,383 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PaloAlto] in [Firewall Service Providers Registry] 2025-06-29 03:36:38,383 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.PaloAltoExternalFirewallElement@456bff44 2025-06-29 03:36:38,384 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PaloAlto] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:38,384 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.PaloAltoExternalFirewallElement@456bff44 2025-06-29 03:36:38,386 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PaloAlto] in [Static Nat Service Providers Registry] 2025-06-29 03:36:38,386 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.PaloAltoExternalFirewallElement@456bff44 2025-06-29 03:36:38,386 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PaloAlto] in [Api Commands Registry] 2025-06-29 03:36:38,386 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.PaloAltoExternalFirewallElement@456bff44 2025-06-29 03:36:38,386 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:38,386 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PaloAlto. 2025-06-29 03:36:38,388 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:38,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [paloalto] in 100 ms 2025-06-29 03:36:38,388 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-network] context. 2025-06-29 03:36:38,388 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-network] context. 2025-06-29 03:36:38,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-network/spring-server-network-context.xml] 2025-06-29 03:36:38,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:38,389 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:38,389 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:38,389 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:38,389 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:38,389 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:38,473 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExternalGuestNetworkGuru 2025-06-29 03:36:38,556 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Ip Deployers Registry] 2025-06-29 03:36:38,556 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,557 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ConfigDrive] in [Dhcp Providers Registry] 2025-06-29 03:36:38,557 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.ConfigDriveNetworkElement@4f0ecc1c 2025-06-29 03:36:38,559 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Dhcp Providers Registry] 2025-06-29 03:36:38,559 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,561 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ControlNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,561 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.ControlNetworkGuru@5a3e14bf 2025-06-29 03:36:38,563 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorageNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,563 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.StorageNetworkGuru@38dab7df 2025-06-29 03:36:38,564 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DirectNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,564 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.DirectNetworkGuru@2f6867d1 2025-06-29 03:36:38,565 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ExternalGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,565 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.ExternalGuestNetworkGuru@3b8bb9a0 2025-06-29 03:36:38,567 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DirectPodBasedNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,567 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.DirectPodBasedNetworkGuru@46b7b94b 2025-06-29 03:36:38,568 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PrivateNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,568 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.PrivateNetworkGuru@d1f1012 2025-06-29 03:36:38,569 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PublicNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,569 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.PublicNetworkGuru@5de49e5a 2025-06-29 03:36:38,570 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [PodBasedNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.PodBasedNetworkGuru@5d3b93b4 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ConfigDrive] in [Network Elements Registry] 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.ConfigDriveNetworkElement@4f0ecc1c 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SecurityGroupProvider] in [Network Elements Registry] 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.SecurityGroupElement@2e53084f 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Network Elements Registry] 2025-06-29 03:36:38,571 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,572 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Firewall Service Providers Registry] 2025-06-29 03:36:38,572 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,574 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:38,574 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,576 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Static Nat Service Providers Registry] 2025-06-29 03:36:38,576 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,577 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:38,578 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,579 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Remote Access V P N Service Provider Registry] 2025-06-29 03:36:38,579 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,579 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VirtualRouter] in [Api Commands Registry] 2025-06-29 03:36:38,579 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.VirtualRouterElement@5ed5ac49 2025-06-29 03:36:38,579 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:38,579 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ControlNetworkGuru. 2025-06-29 03:36:38,580 INFO [c.c.n.g.ControlNetworkGuru] (main:[]) (logid:) Control network setup: cidr=169.254.0.0/16; gateway = 169.254.0.1 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ConfigDrive. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DirectNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ExternalGuestNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DirectPodBasedNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecurityGroupProvider. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VirtualRouter. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PrivateNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PublicNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean PodBasedNetworkGuru. 2025-06-29 03:36:38,580 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-network] in 193 ms 2025-06-29 03:36:38,581 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ssp] context. 2025-06-29 03:36:38,581 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ssp] context. 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ssp/spring-ssp-context.xml] 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:38,581 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ssp] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:38,721 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SspGuestNetworkGuru 2025-06-29 03:36:38,745 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SspGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,745 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.guru.SspGuestNetworkGuru@20de2518 2025-06-29 03:36:38,746 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StratosphereSsp] in [Network Elements Registry] 2025-06-29 03:36:38,746 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.element.SspElement@396e2498 2025-06-29 03:36:38,746 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StratosphereSsp] in [Api Commands Registry] 2025-06-29 03:36:38,746 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.element.SspElement@396e2498 2025-06-29 03:36:38,746 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:38,746 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SspUuidDaoImpl. 2025-06-29 03:36:38,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SspTenantDaoImpl. 2025-06-29 03:36:38,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SspCredentialDaoImpl. 2025-06-29 03:36:38,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StratosphereSsp. 2025-06-29 03:36:38,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SspGuestNetworkGuru. 2025-06-29 03:36:38,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:38,747 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ssp] in 166 ms 2025-06-29 03:36:38,747 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [tungsten] context. 2025-06-29 03:36:38,747 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [tungsten] context. 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/tungsten/spring-tungsten-context.xml] 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:38,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [tungsten] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:38,884 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from TungstenGuestNetworkGuru 2025-06-29 03:36:38,932 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenElement] in [Ip Deployers Registry] 2025-06-29 03:36:38,932 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenElement@214c97e1 2025-06-29 03:36:38,933 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:38,933 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenGuestNetworkGuru@4196a44a 2025-06-29 03:36:38,933 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenElement] in [Network Elements Registry] 2025-06-29 03:36:38,933 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenElement@214c97e1 2025-06-29 03:36:38,935 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenElement] in [Firewall Service Providers Registry] 2025-06-29 03:36:38,935 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenElement@214c97e1 2025-06-29 03:36:38,936 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenElement] in [Port Forwarding Service Providers Registry] 2025-06-29 03:36:38,936 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenElement@214c97e1 2025-06-29 03:36:38,938 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenElement] in [Static Nat Service Providers Registry] 2025-06-29 03:36:38,938 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenElement@214c97e1 2025-06-29 03:36:38,939 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenElement] in [Load Balancing Service Providers Registry] 2025-06-29 03:36:38,939 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenElement@214c97e1 2025-06-29 03:36:38,939 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [TungstenProviderServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:38,939 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.network.tungsten.service.TungstenProviderServiceImpl@6faea62f 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenFabricLBHealthMonitorDaoImpl. 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenProviderServiceImpl. 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenServiceImpl. 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenGuestNetworkGuru. 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean TungstenElement. 2025-06-29 03:36:38,956 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener TungstenElement with id 24 2025-06-29 03:36:38,956 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [tungsten] in 209 ms 2025-06-29 03:36:38,957 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vcs] context. 2025-06-29 03:36:38,957 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vcs] context. 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vcs/spring-vcs-context.xml] 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:38,957 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vcs] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,073 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BrocadeVcsGuestNetworkGuru 2025-06-29 03:36:39,103 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BrocadeVcsGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:39,103 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.BrocadeVcsGuestNetworkGuru@503056df 2025-06-29 03:36:39,104 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BrocadeVcs] in [Network Elements Registry] 2025-06-29 03:36:39,104 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BrocadeVcsElement@5b1965ea 2025-06-29 03:36:39,104 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BrocadeVcs] in [Api Commands Registry] 2025-06-29 03:36:39,104 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.BrocadeVcsElement@5b1965ea 2025-06-29 03:36:39,104 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,104 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BrocadeVcsDaoImpl. 2025-06-29 03:36:39,104 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BrocadeVcsNetworkVlanMappingDaoImpl. 2025-06-29 03:36:39,105 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BrocadeVcs. 2025-06-29 03:36:39,105 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BrocadeVcsGuestNetworkGuru. 2025-06-29 03:36:39,105 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,105 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vcs] in 148 ms 2025-06-29 03:36:39,105 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vmware-network] context. 2025-06-29 03:36:39,106 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vmware-network] context. 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vmware-network/spring-vmware-network-context.xml] 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,106 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-network] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,181 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoNexus1000vVSM] in [Network Elements Registry] 2025-06-29 03:36:39,182 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoNexusVSMElement@4e4f979f 2025-06-29 03:36:39,182 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CiscoNexus1000vVSM] in [Api Commands Registry] 2025-06-29 03:36:39,182 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered com.cloud.network.element.CiscoNexusVSMElement@4e4f979f 2025-06-29 03:36:39,182 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,182 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CiscoNexus1000vVSM. 2025-06-29 03:36:39,182 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vmware-network] in 77 ms 2025-06-29 03:36:39,183 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vxlan] context. 2025-06-29 03:36:39,183 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vxlan] context. 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vxlan/spring-vxlan-context.xml] 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml] 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,183 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vxlan] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,255 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VxlanGuestNetworkGuru 2025-06-29 03:36:39,287 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VxlanGuestNetworkGuru] in [Network Gurus Registry] 2025-06-29 03:36:39,287 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.network.guru.VxlanGuestNetworkGuru@5fbd5e9e 2025-06-29 03:36:39,288 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,288 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VxlanGuestNetworkGuru. 2025-06-29 03:36:39,288 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vxlan] in 105 ms 2025-06-29 03:36:39,288 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-alert-adapter-backend] context. 2025-06-29 03:36:39,288 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-alert-adapter-backend] context. 2025-06-29 03:36:39,288 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-alert-adapter-backend/spring-server-alert-adapter-backend-context.xml] 2025-06-29 03:36:39,289 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,289 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,289 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,289 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,289 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-backend] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,361 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,361 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterAlertAdapter. 2025-06-29 03:36:39,362 INFO [c.c.a.ClusterAlertAdapter] (main:[]) (logid:) Start configuring cluster alert manager : ClusterAlertAdapter 2025-06-29 03:36:39,362 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,362 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-alert-adapter-backend] in 74 ms 2025-06-29 03:36:39,362 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage] context. 2025-06-29 03:36:39,362 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage] context. 2025-06-29 03:36:39,362 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-engine-storage-datamotion-storage-context.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-engine-storage-snapshot-storage-context.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,363 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,503 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorageSystemSnapshotStrategy] in [Snapshot Strategies Registry] 2025-06-29 03:36:39,504 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy@3d8b888f 2025-06-29 03:36:39,505 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CephSnapshotStrategy] in [Snapshot Strategies Registry] 2025-06-29 03:36:39,505 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.CephSnapshotStrategy@2e24c632 2025-06-29 03:36:39,507 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ScaleIOSnapshotStrategy] in [Snapshot Strategies Registry] 2025-06-29 03:36:39,507 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.ScaleIOSnapshotStrategy@430c9347 2025-06-29 03:36:39,509 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DefaultSnapshotStrategy] in [Snapshot Strategies Registry] 2025-06-29 03:36:39,509 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy@2e5f9491 2025-06-29 03:36:39,511 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorageVMSnapshotStrategy] in [Vm Snapshot Strategies Registry] 2025-06-29 03:36:39,511 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.vmsnapshot.StorageVMSnapshotStrategy@7957eec3 2025-06-29 03:36:39,512 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ScaleIOVMSnapshotStrategy] in [Vm Snapshot Strategies Registry] 2025-06-29 03:36:39,512 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.vmsnapshot.ScaleIOVMSnapshotStrategy@38f2eee5 2025-06-29 03:36:39,514 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [DefaultVMSnapshotStrategy] in [Vm Snapshot Strategies Registry] 2025-06-29 03:36:39,514 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.vmsnapshot.DefaultVMSnapshotStrategy@1816701b 2025-06-29 03:36:39,516 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [AncientDataMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:39,516 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.AncientDataMotionStrategy@15f63744 2025-06-29 03:36:39,517 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorageSystemDataMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:39,517 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy@eb89a4b 2025-06-29 03:36:39,519 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HypervStorageMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:39,519 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.HypervStorageMotionStrategy@41de90fa 2025-06-29 03:36:39,520 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KvmNonManagedStorageDataMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:39,520 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.KvmNonManagedStorageDataMotionStrategy@1a50f575 2025-06-29 03:36:39,521 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [XenServerStorageMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:39,522 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy@39b7d031 2025-06-29 03:36:39,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,522 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorageVMSnapshotStrategy. 2025-06-29 03:36:39,523 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ScaleIOVMSnapshotStrategy. 2025-06-29 03:36:39,524 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean DefaultVMSnapshotStrategy. 2025-06-29 03:36:39,525 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage] in 164 ms 2025-06-29 03:36:39,526 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-storage] context. 2025-06-29 03:36:39,526 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [baremetal-storage] context. 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/baremetal-storage/spring-baremetal-storage-context.xml] 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,526 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,527 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,592 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [BareMetalAdapter] in [Template Adapter Registry] 2025-06-29 03:36:39,592 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.manager.BareMetalTemplateAdapter@67a639e3 2025-06-29 03:36:39,592 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,592 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean BareMetalAdapter. 2025-06-29 03:36:39,592 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,592 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [baremetal-storage] in 66 ms 2025-06-29 03:36:39,592 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-alert-adapter-storage] context. 2025-06-29 03:36:39,592 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-alert-adapter-storage] context. 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-alert-adapter-storage/spring-server-alert-adapter-storage-context.xml] 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,593 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-alert-adapter-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,647 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecondaryStorageVmAlertAdapter. 2025-06-29 03:36:39,648 INFO [c.c.a.SecondaryStorageVmAlertAdapter] (main:[]) (logid:) Start configuring secondary storage vm alert manager : SecondaryStorageVmAlertAdapter 2025-06-29 03:36:39,648 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-alert-adapter-storage] in 55 ms 2025-06-29 03:36:39,648 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-storage] context. 2025-06-29 03:36:39,648 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-storage] context. 2025-06-29 03:36:39,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-storage/spring-server-storage-context.xml] 2025-06-29 03:36:39,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,708 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SecondaryStorageVmDefaultAllocator] in [Secondary Storage Vm Allocators Registry] 2025-06-29 03:36:39,708 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.storage.secondary.SecondaryStorageVmDefaultAllocator@1ce55fdf 2025-06-29 03:36:39,709 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,709 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecondaryStorageVmDefaultAllocator. 2025-06-29 03:36:39,709 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-storage] in 61 ms 2025-06-29 03:36:39,709 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-template-adapter] context. 2025-06-29 03:36:39,709 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-template-adapter] context. 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-template-adapter/spring-server-template-adapter-context.xml] 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,709 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-template-adapter] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,773 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [HypervisorAdapter] in [Template Adapter Registry] 2025-06-29 03:36:39,773 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.template.HypervisorTemplateAdapter@1d4e554f 2025-06-29 03:36:39,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean HypervisorAdapter. 2025-06-29 03:36:39,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-template-adapter] in 64 ms 2025-06-29 03:36:39,773 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [sharedfs-provider-storagevm] context. 2025-06-29 03:36:39,773 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [sharedfs-provider-storagevm] context. 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/sharedfs-provider-storagevm/spring-sharedfs-provider-storagevm-context.xml] 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [sharedfs-provider-storagevm] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,827 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from StorageVmSharedFSProvider 2025-06-29 03:36:39,848 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SHAREDFSVM] in [Shared F S Providers Registry] 2025-06-29 03:36:39,848 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.sharedfs.provider.StorageVmSharedFSProvider@2f1661e6 2025-06-29 03:36:39,848 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,848 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SHAREDFSVM. 2025-06-29 03:36:39,848 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,848 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [sharedfs-provider-storagevm] in 75 ms 2025-06-29 03:36:39,848 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-allocator] context. 2025-06-29 03:36:39,849 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-allocator] context. 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-allocator/spring-engine-storage-storage-allocator-context.xml] 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,849 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-allocator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,929 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for org.apache.cloudstack.storage.allocator.GarbageCollectingStoragePoolAllocator@10d29fe 2025-06-29 03:36:39,930 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ZoneWideStoragePoolAllocator] in [Storage Pool Allocators Registry] 2025-06-29 03:36:39,930 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.allocator.ZoneWideStoragePoolAllocator@e4748af 2025-06-29 03:36:39,930 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [LocalStorage] in [Storage Pool Allocators Registry] 2025-06-29 03:36:39,930 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.allocator.LocalStoragePoolAllocator@6689e1b3 2025-06-29 03:36:39,930 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [ClusterScopeStoragePoolAllocator] in [Storage Pool Allocators Registry] 2025-06-29 03:36:39,930 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.allocator.ClusterScopeStoragePoolAllocator@48c7f0d9 2025-06-29 03:36:39,932 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [UserFirst] in [Pod Allocators Registry] 2025-06-29 03:36:39,932 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.agent.manager.allocator.impl.UserConcentratedAllocator@32663c30 2025-06-29 03:36:39,932 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,932 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean GCStorage. 2025-06-29 03:36:39,935 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean UserFirst. 2025-06-29 03:36:39,936 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ZoneWideStoragePoolAllocator. 2025-06-29 03:36:39,936 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LocalStorage. 2025-06-29 03:36:39,936 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean ClusterScopeStoragePoolAllocator. 2025-06-29 03:36:39,936 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,936 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-allocator] in 87 ms 2025-06-29 03:36:39,936 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-image-default] context. 2025-06-29 03:36:39,936 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-image-default] context. 2025-06-29 03:36:39,936 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-image-default/spring-storage-image-default-context.xml] 2025-06-29 03:36:39,937 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,937 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,937 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,937 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,937 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,937 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:39,992 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:NFS 2025-06-29 03:36:39,998 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.CloudStackImageStoreProviderImpl@5a9f6c6c 2025-06-29 03:36:39,998 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:39,999 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-image-default] in 63 ms 2025-06-29 03:36:39,999 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-image-s3] context. 2025-06-29 03:36:39,999 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-image-s3] context. 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-image-s3/spring-storage-image-s3-context.xml] 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:39,999 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-s3] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,069 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:S3 2025-06-29 03:36:40,071 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.S3ImageStoreProviderImpl@e7906af 2025-06-29 03:36:40,072 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,072 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-image-s3] in 73 ms 2025-06-29 03:36:40,072 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-image-swift] context. 2025-06-29 03:36:40,072 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-image-swift] context. 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-image-swift/spring-storage-image-swift-context.xml] 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,072 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-image-swift] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,127 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Swift 2025-06-29 03:36:40,130 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.SwiftImageStoreProviderImpl@31fcad71 2025-06-29 03:36:40,130 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,130 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-image-swift] in 59 ms 2025-06-29 03:36:40,131 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-object-ceph] context. 2025-06-29 03:36:40,131 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-object-ceph] context. 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-object-ceph/spring-storage-object-ceph-context.xml] 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,131 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-ceph] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,191 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Ceph 2025-06-29 03:36:40,201 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.CephObjectStoreProviderImpl@6b828f7b 2025-06-29 03:36:40,201 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,201 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,201 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-object-ceph] in 70 ms 2025-06-29 03:36:40,202 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-object-cloudian] context. 2025-06-29 03:36:40,202 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-object-cloudian] context. 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-object-cloudian/spring-storage-object-cloudian-context.xml] 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,202 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-cloudian] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,278 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Cloudian HyperStore 2025-06-29 03:36:40,284 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.CloudianHyperStoreObjectStoreProviderImpl@7be0330c 2025-06-29 03:36:40,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,285 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-object-cloudian] in 83 ms 2025-06-29 03:36:40,285 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-object-minio] context. 2025-06-29 03:36:40,285 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-object-minio] context. 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-object-minio/spring-storage-object-minio-context.xml] 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,285 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,286 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-minio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,354 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:MinIO 2025-06-29 03:36:40,360 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.MinIOObjectStoreProviderImpl@6d5a8e08 2025-06-29 03:36:40,361 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,361 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-object-minio] in 76 ms 2025-06-29 03:36:40,361 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-object-simulator] context. 2025-06-29 03:36:40,361 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-object-simulator] context. 2025-06-29 03:36:40,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-object-simulator/spring-storage-object-simulator-context.xml] 2025-06-29 03:36:40,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,361 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,362 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,362 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,362 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-object-simulator] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,420 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Simulator 2025-06-29 03:36:40,423 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.SimulatorObjectStoreProviderImpl@1f5e30ca 2025-06-29 03:36:40,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,423 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,423 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-object-simulator] in 62 ms 2025-06-29 03:36:40,423 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-cloudbyte] context. 2025-06-29 03:36:40,423 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-cloudbyte] context. 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-cloudbyte/spring-storage-volume-cloudbyte-context.xml] 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,424 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-cloudbyte] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,496 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:CloudByte 2025-06-29 03:36:40,497 INFO [o.a.c.s.d.p.ElastistorPrimaryDataStoreProvider] (main:[]) (logid:) Will configure elastistor's lifecycle, driver, listener & global configurations. 2025-06-29 03:36:40,517 INFO [o.a.c.s.d.p.ElastistorPrimaryDataStoreProvider] (main:[]) (logid:) Successfully configured elastistor's lifecycle, driver, listener & global configurations. 2025-06-29 03:36:40,518 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.ElastistorPrimaryDataStoreProvider@75ea0909 2025-06-29 03:36:40,518 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [elastistorVolumeApiService] in [Api Commands Registry] 2025-06-29 03:36:40,518 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.util.ElastistorVolumeApiServiceImpl@75332dcf 2025-06-29 03:36:40,519 INFO [o.a.c.s.d.u.ElastistorVolumeApiServiceImpl] (main:[]) (logid:) Commands were registered successfully with elastistor volume api service. [cmdcount:3] 2025-06-29 03:36:40,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean elastistorVolumeApiService. 2025-06-29 03:36:40,519 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-cloudbyte] in 95 ms 2025-06-29 03:36:40,519 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-datera] context. 2025-06-29 03:36:40,519 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-datera] context. 2025-06-29 03:36:40,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml] 2025-06-29 03:36:40,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,519 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,520 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,520 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,520 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-datera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,627 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Datera 2025-06-29 03:36:40,638 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider@7b9e4b0d 2025-06-29 03:36:40,638 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,638 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-datera] in 120 ms 2025-06-29 03:36:40,639 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-default] context. 2025-06-29 03:36:40,639 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-default] context. 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-default/spring-storage-volume-default-context.xml] 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,639 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-default] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,696 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:DefaultPrimary 2025-06-29 03:36:40,705 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.CloudStackPrimaryDataStoreProviderImpl@22c15ebc 2025-06-29 03:36:40,705 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,705 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-default] in 66 ms 2025-06-29 03:36:40,705 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-flasharray] context. 2025-06-29 03:36:40,705 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-flasharray] context. 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-flasharray/spring-storage-volume-flasharray-context.xml] 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,705 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,706 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-flasharray] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,760 INFO [o.a.c.s.d.p.FlashArrayPrimaryDatastoreProviderImpl] (main:[]) (logid:) Creating Flash Array 2025-06-29 03:36:40,765 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Flash Array 2025-06-29 03:36:40,765 INFO [o.a.c.s.d.p.FlashArrayPrimaryDatastoreProviderImpl] (main:[]) (logid:) Configuring Flash Array 2025-06-29 03:36:40,773 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.FlashArrayPrimaryDatastoreProviderImpl@417df2ed 2025-06-29 03:36:40,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-flasharray] in 68 ms 2025-06-29 03:36:40,773 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-linstor] context. 2025-06-29 03:36:40,773 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-linstor] context. 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloud-plugin-storage-volume-linstor-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-linstor/spring-storage-volume-linstor-context.xml] 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-linstor] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,836 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from LinstorConfigurationManager 2025-06-29 03:36:40,852 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Linstor 2025-06-29 03:36:40,865 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.LinstorPrimaryDatastoreProviderImpl@3f36b79d 2025-06-29 03:36:40,867 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [LinstorVMSnapshotStrategy] in [Vm Snapshot Strategies Registry] 2025-06-29 03:36:40,867 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.LinstorVMSnapshotStrategy@1501b99a 2025-06-29 03:36:40,867 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,867 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean LinstorVMSnapshotStrategy. 2025-06-29 03:36:40,868 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-linstor] in 96 ms 2025-06-29 03:36:40,869 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-primera] context. 2025-06-29 03:36:40,869 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-primera] context. 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-primera/spring-storage-volume-primera-context.xml] 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,869 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-primera] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,917 INFO [o.a.c.s.d.p.PrimeraPrimaryDatastoreProviderImpl] (main:[]) (logid:) Creating Primera 2025-06-29 03:36:40,920 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:Primera 2025-06-29 03:36:40,920 INFO [o.a.c.s.d.p.PrimeraPrimaryDatastoreProviderImpl] (main:[]) (logid:) Configuring Primera 2025-06-29 03:36:40,920 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.PrimeraPrimaryDatastoreProviderImpl@1a82c5b5 2025-06-29 03:36:40,921 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,921 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-primera] in 52 ms 2025-06-29 03:36:40,921 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-scaleio] context. 2025-06-29 03:36:40,921 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-scaleio] context. 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-scaleio/spring-storage-volume-scaleio-context.xml] 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,921 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-scaleio] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:40,974 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ScaleIOSDCManagerImpl 2025-06-29 03:36:40,987 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:PowerFlex 2025-06-29 03:36:40,997 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.ScaleIOPrimaryDatastoreProvider@1b7a93b5 2025-06-29 03:36:40,997 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:40,997 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-scaleio] in 77 ms 2025-06-29 03:36:40,998 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-solidfire] context. 2025-06-29 03:36:40,998 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-solidfire] context. 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-solidfire/spring-storage-volume-solidfire-context.xml] 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:40,998 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-solidfire] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,063 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:SolidFireShared 2025-06-29 03:36:41,071 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.SolidFireSharedPrimaryDataStoreProvider@6bd8beab 2025-06-29 03:36:41,072 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:SolidFire 2025-06-29 03:36:41,081 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.SolidFirePrimaryDataStoreProvider@14446fec 2025-06-29 03:36:41,081 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,081 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-solidfire] in 83 ms 2025-06-29 03:36:41,082 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [storage-volume-storpool] context. 2025-06-29 03:36:41,082 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [storage-volume-storpool] context. 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloud-plugin-storage-volume-storpool-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-storpool/spring-storage-volume-storpool-context.xml] 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [storage-volume-storpool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,159 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from StorPoolConfigurationManager 2025-06-29 03:36:41,248 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from StorPoolAbandonObjectsCollector 2025-06-29 03:36:41,274 DEBUG [o.a.c.s.d.p.DataStoreProviderManagerImpl] (main:[]) (logid:) registering data store provider:StorPool 2025-06-29 03:36:41,288 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.datastore.provider.StorPoolPrimaryDataStoreProvider@c119654 2025-06-29 03:36:41,291 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorPoolSnapshotStrategy] in [Snapshot Strategies Registry] 2025-06-29 03:36:41,291 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.StorPoolSnapshotStrategy@6db89763 2025-06-29 03:36:41,293 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorPoolVMSnapshotStrategy] in [Vm Snapshot Strategies Registry] 2025-06-29 03:36:41,293 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.snapshot.StorPoolVMSnapshotStrategy@11b7e565 2025-06-29 03:36:41,295 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [StorPoolDataMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:41,295 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.StorPoolDataMotionStrategy@5ea7b4ab 2025-06-29 03:36:41,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorPoolAbandonObjectsCollector. 2025-06-29 03:36:41,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorPoolStatsCollector. 2025-06-29 03:36:41,295 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean StorPoolVMSnapshotStrategy. 2025-06-29 03:36:41,297 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [storage-volume-storpool] in 216 ms 2025-06-29 03:36:41,298 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vmware-storage] context. 2025-06-29 03:36:41,298 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vmware-storage] context. 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vmware-storage/spring-vmware-storage-context.xml] 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage/spring-lifecycle-storage-context-inheritable.xml] 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,298 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,299 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-storage] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,394 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VmwareStorageMotionStrategy] in [Data Motion Strategies Registry] 2025-06-29 03:36:41,394 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy@25e92583 2025-06-29 03:36:41,394 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,394 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vmware-storage] in 97 ms 2025-06-29 03:36:41,395 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [cluster] context. 2025-06-29 03:36:41,395 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [cluster] context. 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cluster] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/cluster/spring-core-lifecycle-cluster-context-inheritable.xml] 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cluster] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cluster] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cluster] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cluster] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,395 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [cluster] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,454 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,454 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [cluster] in 59 ms 2025-06-29 03:36:41,454 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [balanced] context. 2025-06-29 03:36:41,454 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [balanced] context. 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/balanced/spring-balanced-context.xml] 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/cluster/spring-core-lifecycle-cluster-context-inheritable.xml] 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,455 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [balanced] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,536 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [balanced] in [Cluster Drs Algorithm Registry] 2025-06-29 03:36:41,536 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.cluster.Balanced@3e009dca 2025-06-29 03:36:41,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean balanced. 2025-06-29 03:36:41,536 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [balanced] in 82 ms 2025-06-29 03:36:41,537 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [condensed] context. 2025-06-29 03:36:41,537 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [condensed] context. 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/condensed/spring-condensed-context.xml] 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/cluster/spring-core-lifecycle-cluster-context-inheritable.xml] 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,537 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [condensed] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,602 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [condensed] in [Cluster Drs Algorithm Registry] 2025-06-29 03:36:41,602 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.cluster.Condensed@ea29f27 2025-06-29 03:36:41,602 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,602 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean condensed. 2025-06-29 03:36:41,602 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,602 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [condensed] in 65 ms 2025-06-29 03:36:41,602 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [discoverer] context. 2025-06-29 03:36:41,602 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [discoverer] context. 2025-06-29 03:36:41,603 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,603 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,603 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,603 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,603 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,603 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,648 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,648 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [discoverer] in 46 ms 2025-06-29 03:36:41,648 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-discoverer] context. 2025-06-29 03:36:41,648 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [baremetal-discoverer] context. 2025-06-29 03:36:41,648 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/baremetal-discoverer/spring-baremetal-discoverer-context.xml] 2025-06-29 03:36:41,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,649 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [baremetal-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,709 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Bare Metal Agent] in [Resource Discoverers Registry] 2025-06-29 03:36:41,709 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.baremetal.manager.BareMetalDiscoverer@258abd37 2025-06-29 03:36:41,709 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,709 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Bare Metal Agent. 2025-06-29 03:36:41,709 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [baremetal-discoverer] in 62 ms 2025-06-29 03:36:41,710 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [hyperv-discoverer] context. 2025-06-29 03:36:41,710 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [hyperv-discoverer] context. 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/hyperv-discoverer/spring-hyperv-discoverer-context.xml] 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,710 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [hyperv-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,773 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Hyper-V Agent] in [Resource Discoverers Registry] 2025-06-29 03:36:41,773 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer@5048b102 2025-06-29 03:36:41,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Hyper-V Agent. 2025-06-29 03:36:41,773 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener HypervServerDiscoverer with id 25 2025-06-29 03:36:41,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,773 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [hyperv-discoverer] in 63 ms 2025-06-29 03:36:41,773 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ovm-discoverer] context. 2025-06-29 03:36:41,773 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ovm-discoverer] context. 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ovm-discoverer/spring-ovm-discoverer-context.xml] 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,774 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,826 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovm Discover] in [Resource Discoverers Registry] 2025-06-29 03:36:41,826 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.ovm.hypervisor.OvmDiscoverer@2a336787 2025-06-29 03:36:41,826 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,826 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ovm Discover. 2025-06-29 03:36:41,827 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,827 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ovm-discoverer] in 53 ms 2025-06-29 03:36:41,827 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ovm3-discoverer] context. 2025-06-29 03:36:41,827 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ovm3-discoverer] context. 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ovm3-discoverer/spring-ovm3-discoverer-context.xml] 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,828 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ovm3-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,888 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Ovm3Discover] in [Resource Discoverers Registry] 2025-06-29 03:36:41,888 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.ovm3.resources.Ovm3Discoverer@63d3006a 2025-06-29 03:36:41,888 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,889 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Ovm3Discover. 2025-06-29 03:36:41,889 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,889 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ovm3-discoverer] in 61 ms 2025-06-29 03:36:41,889 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [secondary-storage-discoverer] context. 2025-06-29 03:36:41,889 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [secondary-storage-discoverer] context. 2025-06-29 03:36:41,889 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/secondary-storage-discoverer/spring-secondary-storage-discoverer-context.xml] 2025-06-29 03:36:41,889 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,889 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,889 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,889 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,890 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,890 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [secondary-storage-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:41,943 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [SecondaryStorage] in [Resource Discoverers Registry] 2025-06-29 03:36:41,943 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.storage.resource.SecondaryStorageDiscoverer@33db8d3a 2025-06-29 03:36:41,943 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:41,943 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean SecondaryStorage. 2025-06-29 03:36:41,943 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:41,943 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [secondary-storage-discoverer] in 54 ms 2025-06-29 03:36:41,943 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-discoverer] context. 2025-06-29 03:36:41,943 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [server-discoverer] context. 2025-06-29 03:36:41,943 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/server-discoverer/spring-server-discoverer-context.xml] 2025-06-29 03:36:41,944 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:41,944 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:41,944 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:41,944 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:41,944 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:41,944 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [server-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [KVM Agent] in [Resource Discoverers Registry] 2025-06-29 03:36:42,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.kvm.discoverer.KvmServerDiscoverer@23927f47 2025-06-29 03:36:42,023 WARN [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Bean registration failed for com.cloud.resource.DummyHostDiscoverer@2b8acd9d 2025-06-29 03:36:42,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [CustomHW Agent] in [Resource Discoverers Registry] 2025-06-29 03:36:42,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.discoverer.CustomServerDiscoverer@13705509 2025-06-29 03:36:42,023 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [Lxc Discover] in [Resource Discoverers Registry] 2025-06-29 03:36:42,023 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.kvm.discoverer.LxcServerDiscoverer@3a7f08c0 2025-06-29 03:36:42,024 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,024 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean KVM Agent. 2025-06-29 03:36:42,028 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener KvmServerDiscoverer with id 26 2025-06-29 03:36:42,028 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean dummyHostDiscoverer. 2025-06-29 03:36:42,028 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean CustomHW Agent. 2025-06-29 03:36:42,031 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener CustomServerDiscoverer with id 27 2025-06-29 03:36:42,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean Lxc Discover. 2025-06-29 03:36:42,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener LxcServerDiscoverer with id 28 2025-06-29 03:36:42,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,035 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [server-discoverer] in 92 ms 2025-06-29 03:36:42,035 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vmware-discoverer] context. 2025-06-29 03:36:42,035 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [vmware-discoverer] context. 2025-06-29 03:36:42,035 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/vmware-discoverer/spring-vmware-discoverer-context.xml] 2025-06-29 03:36:42,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:42,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [vmware-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,086 INFO [c.c.h.v.VmwareServerDiscoverer] (main:[]) (logid:) VmwareServerDiscoverer is constructed 2025-06-29 03:36:42,094 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VMware Discover] in [Resource Discoverers Registry] 2025-06-29 03:36:42,094 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.vmware.VmwareServerDiscoverer@6c8c59b3 2025-06-29 03:36:42,095 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,095 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean VMware Discover. 2025-06-29 03:36:42,095 INFO [c.c.h.v.VmwareServerDiscoverer] (main:[]) (logid:) Configure VmwareServerDiscoverer, discover name: VMware Discover 2025-06-29 03:36:42,112 INFO [c.c.h.v.VmwareServerDiscoverer] (main:[]) (logid:) VmwareServerDiscoverer has been successfully configured 2025-06-29 03:36:42,112 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,112 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [vmware-discoverer] in 77 ms 2025-06-29 03:36:42,112 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [xenserver-discoverer] context. 2025-06-29 03:36:42,112 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [xenserver-discoverer] context. 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/xenserver-discoverer/spring-xenserver-discoverer-context.xml] 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/discoverer/spring-core-lifecycle-discoverer-context-inheritable.xml] 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,113 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [xenserver-discoverer] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,189 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [XCP Agent] in [Resource Discoverers Registry] 2025-06-29 03:36:42,189 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer@5b0480a3 2025-06-29 03:36:42,189 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,189 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean XCP Agent. 2025-06-29 03:36:42,203 INFO [c.c.h.x.r.XenServerConnectionPool] (main:[]) (logid:) XenServer Connection Pool Configs: sleep.interval.on.error=10000 2025-06-29 03:36:42,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Registering listener XcpServerDiscoverer with id 29 2025-06-29 03:36:42,210 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [xenserver-discoverer] in 98 ms 2025-06-29 03:36:42,211 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [event] context. 2025-06-29 03:36:42,211 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [event] context. 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [event] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/event/spring-core-lifecycle-event-context-inheritable.xml] 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [event] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [event] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [event] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [event] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,211 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [event] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,257 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,257 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,257 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [event] in 46 ms 2025-06-29 03:36:42,257 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [webhook] context. 2025-06-29 03:36:42,257 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [webhook] context. 2025-06-29 03:36:42,257 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/webhook/spring-event-webhook-context.xml] 2025-06-29 03:36:42,257 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/event/spring-core-lifecycle-event-context-inheritable.xml] 2025-06-29 03:36:42,257 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,257 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,258 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,258 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,258 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [webhook] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,383 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from WebhookServiceImpl 2025-06-29 03:36:42,452 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [webhookEventBus] in [Event Bus Registry] 2025-06-29 03:36:42,452 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.mom.webhook.WebhookEventBus@34bc32be 2025-06-29 03:36:42,452 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [WebhookApiServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:42,452 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.mom.webhook.WebhookApiServiceImpl@7249d9ce 2025-06-29 03:36:42,452 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [WebhookServiceImpl] in [Api Commands Registry] 2025-06-29 03:36:42,452 DEBUG [o.a.c.s.l.r.PluggableServiceLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.mom.webhook.WebhookServiceImpl@4a0e3df6 2025-06-29 03:36:42,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean WebhookDeliveryDaoImpl. 2025-06-29 03:36:42,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean WebhookDeliveryJoinDaoImpl. 2025-06-29 03:36:42,453 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean WebhookDaoImpl. 2025-06-29 03:36:42,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean WebhookJoinDaoImpl. 2025-06-29 03:36:42,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean webhookEventBus. 2025-06-29 03:36:42,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean WebhookApiServiceImpl. 2025-06-29 03:36:42,457 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean WebhookServiceImpl. 2025-06-29 03:36:42,459 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,459 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [webhook] in 202 ms 2025-06-29 03:36:42,459 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [outofbandmanagement] context. 2025-06-29 03:36:42,459 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [outofbandmanagement] context. 2025-06-29 03:36:42,459 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [outofbandmanagement] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/outofbandmanagement/spring-core-lifecycle-outofbandmanagement-context-inheritable.xml] 2025-06-29 03:36:42,459 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [outofbandmanagement] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,459 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [outofbandmanagement] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,459 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [outofbandmanagement] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,459 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [outofbandmanagement] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,460 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [outofbandmanagement] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,506 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [outofbandmanagement] in 48 ms 2025-06-29 03:36:42,507 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ipmitool] context. 2025-06-29 03:36:42,507 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [ipmitool] context. 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/ipmitool/spring-ipmitool-context.xml] 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/outofbandmanagement/spring-core-lifecycle-outofbandmanagement-context-inheritable.xml] 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,507 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [ipmitool] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,607 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from IpmitoolOutOfBandManagementDriver 2025-06-29 03:36:42,629 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [IPMITOOL] in [Out Of Band Management Drivers Registry] 2025-06-29 03:36:42,629 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.outofbandmanagement.driver.ipmitool.IpmitoolOutOfBandManagementDriver@38c607b2 2025-06-29 03:36:42,629 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,630 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean IPMITOOL. 2025-06-29 03:36:42,630 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [ipmitool] in 123 ms 2025-06-29 03:36:42,630 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [nested-cloudstack] context. 2025-06-29 03:36:42,630 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [nested-cloudstack] context. 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/nested-cloudstack/spring-nested-cloudstack-context.xml] 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/outofbandmanagement/spring-core-lifecycle-outofbandmanagement-context-inheritable.xml] 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,630 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [nested-cloudstack] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,678 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [NESTEDCLOUDSTACK] in [Out Of Band Management Drivers Registry] 2025-06-29 03:36:42,678 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.outofbandmanagement.driver.nestedcloudstack.NestedCloudStackOutOfBandManagementDriver@3aa8cbcb 2025-06-29 03:36:42,678 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,678 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean NESTEDCLOUDSTACK. 2025-06-29 03:36:42,678 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [nested-cloudstack] in 48 ms 2025-06-29 03:36:42,678 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [redfish] context. 2025-06-29 03:36:42,678 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [redfish] context. 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/redfish/spring-redfish-context.xml] 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/outofbandmanagement/spring-core-lifecycle-outofbandmanagement-context-inheritable.xml] 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml] 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2025-06-29 03:36:42,678 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2025-06-29 03:36:42,679 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [redfish] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.21.0.0-SNAPSHOT.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2025-06-29 03:36:42,723 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RedfishOutOfBandManagementDriver 2025-06-29 03:36:42,745 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [REDFISH] in [Out Of Band Management Drivers Registry] 2025-06-29 03:36:42,745 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.outofbandmanagement.driver.redfish.RedfishOutOfBandManagementDriver@44f9e70f 2025-06-29 03:36:42,745 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:42,746 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean REDFISH. 2025-06-29 03:36:42,746 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:42,746 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loaded module context [redfish] in 68 ms 2025-06-29 03:36:42,746 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [bootstrap] context. 2025-06-29 03:36:42,747 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Could not get module [bootstrap] context bean. 2025-06-29 03:36:42,747 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [system] context. 2025-06-29 03:36:42,747 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [system]. 2025-06-29 03:36:42,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:42,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VersionDaoImpl. 2025-06-29 03:36:42,747 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,748 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:42,748 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [core] context. 2025-06-29 03:36:42,748 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [core]. 2025-06-29 03:36:42,748 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:42,748 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,749 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,749 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,752 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,755 INFO [c.c.h.h.m.HypervManagerImpl] (main:[]) (logid:) Cleanup mounted mount points used in previous session 2025-06-29 03:36:42,755 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,755 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PublicIpQuarantineDaoImpl. 2025-06-29 03:36:42,756 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PhysicalNetworkDaoImpl. 2025-06-29 03:36:42,756 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainDetailsDaoImpl. 2025-06-29 03:36:42,756 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RoleDaoImpl. 2025-06-29 03:36:42,756 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostDetailsDaoImpl. 2025-06-29 03:36:42,756 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StoragePoolWorkDaoImpl. 2025-06-29 03:36:42,757 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConfigurationGroupDaoImpl. 2025-06-29 03:36:42,757 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean GuestOsDetailsDaoImpl. 2025-06-29 03:36:42,757 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMNetworkMapDaoImpl. 2025-06-29 03:36:42,757 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceCountDaoImpl. 2025-06-29 03:36:42,757 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmProfileDetailsDaoImpl. 2025-06-29 03:36:42,757 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotJoinDaoImpl. 2025-06-29 03:36:42,758 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmDeployAsIsDetailsDaoImpl. 2025-06-29 03:36:42,758 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMSnapshotDetailsDaoImpl. 2025-06-29 03:36:42,758 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PodVlanDaoImpl. 2025-06-29 03:36:42,758 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HypervisorCapabilitiesDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageSecurityGroupDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VnfTemplateDetailsDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PhysicalNetworkIsolationMethodDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmGroupStatisticsDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataCenterDetailsDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UcsBladeDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HAConfigDaoImpl. 2025-06-29 03:36:42,759 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerStatusDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserDataDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainRouterJoinDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AccountVlanMapDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SyncQueueDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ASNumberDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConfigurationSubGroupDaoImpl. 2025-06-29 03:36:42,760 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StoragePoolAndAccessGroupMapDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CommandExecLogDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TemplateJoinDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean OutOfBandManagementDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmGroupVmMapDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserStatsLogDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceIconDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NicDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImageStoreJoinDaoImpl. 2025-06-29 03:36:42,761 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotPolicyDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotZoneDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcGatewayDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecurityGroupRulesDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkOfferingDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PlannerHostReservationDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirewallRulesDaoImpl. 2025-06-29 03:36:42,762 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterVSMMapDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VlanDetailsDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkDetailsDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BgpPeerNetworkMapDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AffinityGroupDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ServiceOfferingJoinDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DcDetailsDaoImpl. 2025-06-29 03:36:42,763 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerHostPeerDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageLoadBalancerPolicyDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirewallRuleDetailsDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LegacyZoneDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AsyncJobJoinMapDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Ipv4GuestSubnetNetworkMapDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LBHealthCheckPolicyDetailsDaoImpl. 2025-06-29 03:36:42,764 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackupScheduleDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectRoleDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcGatewayDetailsDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcOfferingDetailsDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageJobDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpnUserDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LBStickinessPolicyDetailsDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CiscoNexusVSMDeviceDaoImpl. 2025-06-29 03:36:42,765 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmTemplateDaoImpl. 2025-06-29 03:36:42,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcOfferingDaoImpl. 2025-06-29 03:36:42,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainRouterDaoImpl. 2025-06-29 03:36:42,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectDaoImpl. 2025-06-29 03:36:42,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean InstanceGroupVMMapDaoImpl. 2025-06-29 03:36:42,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BucketStatisticsDaoImpl. 2025-06-29 03:36:42,766 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeStatsDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageEventDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PrimaryDataStoreDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AsyncJobDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BgpPeerDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageBackupDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageEventDetailsDaoImpl. 2025-06-29 03:36:42,767 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SharedFSJoinDaoImpl. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectAccountDaoImpl. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcOfferingServiceMapDaoImpl. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMTemplateDetailsDaoImpl. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CounterDaoImpl. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean vnet Alloc. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeDetailsDaoImpl. 2025-06-29 03:36:42,768 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BaremetalDhcpDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMTemplateZoneDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PortProfileDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageNetworkOfferingDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLListDetailsDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMComputeTagDaoImpl. 2025-06-29 03:36:42,769 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataCenterJoinDaoImpl. 2025-06-29 03:36:42,770 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsagePortForwardingRuleDaoImpl. 2025-06-29 03:36:42,770 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LoadBalancerDaoImpl. 2025-06-29 03:36:42,770 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HighAvailabilityDaoImpl. 2025-06-29 03:36:42,770 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmCloneSettingDaoImpl. 2025-06-29 03:36:42,770 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SslCertDaoImpl. 2025-06-29 03:36:42,770 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirewallRulesCidrsDaoImpl. 2025-06-29 03:36:42,771 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmDiskStatisticsDaoImpl. 2025-06-29 03:36:42,771 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserAccountJoinDaoImpl. 2025-06-29 03:36:42,771 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMTemplateDaoImpl. 2025-06-29 03:36:42,771 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackupDaoImpl. 2025-06-29 03:36:42,771 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmGroupDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecurityGroupWorkDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean GuestOSHypervisorDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LoadBalancerVMMapDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TungstenProviderDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteVpnConnectionDetailsDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AsyncJobJournalDaoImpl. 2025-06-29 03:36:42,772 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean InstanceGroupJoinDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmwareDatacenterZoneMapDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RouterNetworkDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteCustomerGatewayDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean InstanceGroupDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotScheduleDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmJoinDaoImpl. 2025-06-29 03:36:42,773 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmwareDatacenterDaoImpl. 2025-06-29 03:36:42,774 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean GlobalLoadBalancerLbRuleMapDaoImpl. 2025-06-29 03:36:42,774 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerHostPeerJoinDaoImpl. 2025-06-29 03:36:42,774 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkPermissionDaoImpl. 2025-06-29 03:36:42,774 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageIPAddressDaoImpl. 2025-06-29 03:36:42,774 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLDaoImpl. 2025-06-29 03:36:42,774 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkOfferingServiceMapDaoImpl. 2025-06-29 03:36:42,775 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RouterHealthCheckResultDaoImpl. 2025-06-29 03:36:42,775 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PhysicalNetworkServiceProviderDaoImpl. 2025-06-29 03:36:42,775 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceLimitDaoImpl. 2025-06-29 03:36:42,775 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotDetailsDaoImpl. 2025-06-29 03:36:42,775 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConditionDaoImpl. 2025-06-29 03:36:42,775 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ServiceOfferingDetailsDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AccountDetailsDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SSHKeyPairDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmDetailsDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImageStoreDetailsDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ObjectInDataStoreDaoImpl. 2025-06-29 03:36:42,776 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotDataStoreDaoImpl. 2025-06-29 03:36:42,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PrivateIpDaoImpl. 2025-06-29 03:36:42,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConsoleProxyDaoImpl. 2025-06-29 03:36:42,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DiskOfferingDetailsDaoImpl. 2025-06-29 03:36:42,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerHostDaoImpl. 2025-06-29 03:36:42,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMRootDiskTagDaoImpl. 2025-06-29 03:36:42,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PodVlanMapDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PrimaryDataStoreDetailsDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageVpcDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AccountJoinDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TungstenSecurityGroupRuleDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VirtualRouterProviderDaoImpl. 2025-06-29 03:36:42,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectRolePermissionsDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserAccountDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackupOfferingDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DetailsDao. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkRuleConfigDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PhysicalNetworkTagDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean IPAddressDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PassphraseDaoImpl. 2025-06-29 03:36:42,779 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainVlanMapDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirewallRulesDcidrsDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostGpuGroupsDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DirectDownloadCertificateHostMapDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteVpnGatewayDetailsDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserStatisticsDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TemplateDataStoreDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NsxProviderDaoImpl. 2025-06-29 03:36:42,780 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Ip Alloc. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteCustomerGatewayDetailsDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMTemplatePoolDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmGroupPolicyMapDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EngineClusterDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecondaryStorageVmDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImageStoreDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AffinityGroupDomainMapDaoImpl. 2025-06-29 03:36:42,781 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UcsManagerDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ASNumberRangeDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CapacityDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EventDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RemoteAccessVpnDetailsDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataCenterGuestIpv6PrefixDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RegionDaoImpl. 2025-06-29 03:36:42,782 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkServiceMapDaoImpl. 2025-06-29 03:36:42,783 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMReservationDaoImpl. 2025-06-29 03:36:42,783 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecurityGroupRuleDaoImpl. 2025-06-29 03:36:42,783 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ReconcileCommandDaoImpl. 2025-06-29 03:36:42,784 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BgpPeerDetailsDaoImpl. 2025-06-29 03:36:42,784 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BaremetalRctDaoImpl. 2025-06-29 03:36:42,784 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostTransferMapDaoImpl. 2025-06-29 03:36:42,784 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ItWorkDaoImpl. 2025-06-29 03:36:42,785 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcOfferingJoinDaoImpl. 2025-06-29 03:36:42,785 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImageStoreObjectDownloadDaoImpl. 2025-06-29 03:36:42,785 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecurityGroupJoinDaoImpl. 2025-06-29 03:36:42,785 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean GlobalLoadBalancerDaoImpl. 2025-06-29 03:36:42,785 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LoadBalancerCertMapDaoImpl. 2025-06-29 03:36:42,786 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DirectDownloadCertificateDaoImpl. 2025-06-29 03:36:42,786 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScalePolicyConditionMapDaoImpl. 2025-06-29 03:36:42,786 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BucketDaoImpl. 2025-06-29 03:36:42,786 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmStatsDaoImpl. 2025-06-29 03:36:42,786 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeDataStoreDaoImpl. 2025-06-29 03:36:42,786 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectInvitationJoinDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LBStickinessPolicyDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectJoinDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VlanDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StaticRouteDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostTagsDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageVolumeDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkExternalLoadBalancerDaoImpl. 2025-06-29 03:36:42,787 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DiskOfferingDaoImpl. 2025-06-29 03:36:42,788 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkAccountDaoImpl. 2025-06-29 03:36:42,788 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VsphereStoragePolicyDaoImpl. 2025-06-29 03:36:42,788 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageDaoImpl. 2025-06-29 03:36:42,788 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AccountDaoImpl. 2025-06-29 03:36:42,788 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserDaoImpl. 2025-06-29 03:36:42,788 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceTagJoinDaoImpl. 2025-06-29 03:36:42,789 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean OvsProviderDaoImpl. 2025-06-29 03:36:42,789 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectInvitationDaoImpl. 2025-06-29 03:36:42,789 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainDaoImpl. 2025-06-29 03:36:42,789 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CrlDaoImpl. 2025-06-29 03:36:42,789 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TemplateDeployAsIsDetailsDaoImpl. 2025-06-29 03:36:42,789 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserDetailsDaoImpl. 2025-06-29 03:36:42,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EngineDataCenterDaoImpl. 2025-06-29 03:36:42,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageVPNUserDaoImpl. 2025-06-29 03:36:42,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLItemDaoImpl. 2025-06-29 03:36:42,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMScheduleDaoImpl. 2025-06-29 03:36:42,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmDaoImpl. 2025-06-29 03:36:42,790 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkExternalFirewallDaoImpl. 2025-06-29 03:36:42,791 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostPodDaoImpl. 2025-06-29 03:36:42,791 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AccountGuestVlanMapDaoImpl. 2025-06-29 03:36:42,791 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteVpnGatewayDaoImpl. 2025-06-29 03:36:42,791 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Ipv6GuestPrefixSubnetNetworkMapDaoImpl. 2025-06-29 03:36:42,791 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmGroupDetailsDaoImpl. 2025-06-29 03:36:42,791 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SyncQueueItemDaoImpl. 2025-06-29 03:36:42,792 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RemoteAccessVpnDaoImpl. 2025-06-29 03:36:42,792 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AffinityGroupVMMapDaoImpl. 2025-06-29 03:36:42,792 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BaremetalPxeDaoImpl. 2025-06-29 03:36:42,792 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageVmDiskDaoImpl. 2025-06-29 03:36:42,792 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleVmProfileDaoImpl. 2025-06-29 03:36:42,792 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcServiceMapDaoImpl. 2025-06-29 03:36:42,793 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotDaoImpl. 2025-06-29 03:36:42,793 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EventJoinDaoImpl. 2025-06-29 03:36:42,793 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterDetailsDaoImpl. 2025-06-29 03:36:42,793 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLItemCidrsDaoImpl. 2025-06-29 03:36:42,793 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMScheduledJobDaoImpl. 2025-06-29 03:36:42,793 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLItemDetailsDaoImpl. 2025-06-29 03:36:42,794 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ObjectStoreDetailsDaoImpl. 2025-06-29 03:36:42,794 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NicIpAliasDaoImpl. 2025-06-29 03:36:42,794 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataCenterLinkLocalIpAddressDaoImpl. 2025-06-29 03:36:42,794 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerJoinDaoImpl. 2025-06-29 03:36:42,795 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AlertDaoImpl. 2025-06-29 03:36:42,795 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ReservationDaoImpl. 2025-06-29 03:36:42,795 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VnfTemplateNicDaoImpl. 2025-06-29 03:36:42,795 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecondaryStorageHeuristicDaoImpl. 2025-06-29 03:36:42,796 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkOfferingDetailsDaoImpl. 2025-06-29 03:36:42,796 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ObjectStoreDaoImpl. 2025-06-29 03:36:42,796 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AnnotationDaoImpl. 2025-06-29 03:36:42,796 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AffinityGroupJoinDaoImpl. 2025-06-29 03:36:42,796 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMSnapshotDaoImpl. 2025-06-29 03:36:42,797 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmRulesetLogDaoImpl. 2025-06-29 03:36:42,797 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EngineHostDaoImpl. 2025-06-29 03:36:42,797 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StoragePoolJoinDaoImpl. 2025-06-29 03:36:42,797 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NicExtraDhcpOptionDaoImpl. 2025-06-29 03:36:42,797 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterDrsPlanMigrationDaoImpl. 2025-06-29 03:36:42,797 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DedicatedResourceDaoImpl. 2025-06-29 03:36:42,798 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotPolicyDetailsDaoImpl. 2025-06-29 03:36:42,798 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecurityGroupDaoImpl. 2025-06-29 03:36:42,798 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean GuestOSCategoryDaoImpl. 2025-06-29 03:36:42,798 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean MonitoringServiceDaoImpl. 2025-06-29 03:36:42,798 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StoragePoolHostDaoImpl. 2025-06-29 03:36:42,798 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostJoinDaoImpl. 2025-06-29 03:36:42,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeReservationDaoImpl. 2025-06-29 03:36:42,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LBHealthCheckPolicyDaoImpl. 2025-06-29 03:36:42,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataCenterDaoImpl. 2025-06-29 03:36:42,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StorageNetworkIpAddressDaoImpl. 2025-06-29 03:36:42,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserIpAddressDetailsDaoImpl. 2025-06-29 03:36:42,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PortableIpDaoImpl. 2025-06-29 03:36:42,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmWorkJobDaoImpl. 2025-06-29 03:36:42,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserIpv6AddressDaoImpl. 2025-06-29 03:36:42,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostTagsDaoImpl. 2025-06-29 03:36:42,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteVpnConnectionDaoImpl. 2025-06-29 03:36:42,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageNetworksDaoImpl. 2025-06-29 03:36:42,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageStorageDaoImpl. 2025-06-29 03:36:42,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageNetworkDaoImpl. 2025-06-29 03:36:42,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcDaoImpl. 2025-06-29 03:36:42,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StoragePoolTagsDaoImpl. 2025-06-29 03:36:42,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DiskOfferingJoinDaoImpl. 2025-06-29 03:36:42,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMInstanceDaoImpl. 2025-06-29 03:36:42,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeJoinDaoImpl. 2025-06-29 03:36:42,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMEntityDaoImpl. 2025-06-29 03:36:42,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SharedFSDaoImpl. 2025-06-29 03:36:42,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean InlineLoadBalancerNicMapDaoImpl. 2025-06-29 03:36:42,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PhysicalNetworkTrafficTypeDaoImpl. 2025-06-29 03:36:42,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkOfferingJoinDaoImpl. 2025-06-29 03:36:42,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LaunchPermissionDaoImpl. 2025-06-29 03:36:42,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScalePolicyDaoImpl. 2025-06-29 03:36:42,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TungstenGuestNetworkIpAddressDaoImpl. 2025-06-29 03:36:42,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkDaoImpl. 2025-06-29 03:36:42,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PortForwardingRulesDaoImpl. 2025-06-29 03:36:42,804 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkDaoImpl. 2025-06-29 03:36:42,804 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ExternalFirewallDeviceDaoImpl. 2025-06-29 03:36:42,804 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ServiceOfferingDaoImpl. 2025-06-29 03:36:42,804 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApplicationLoadBalancerRuleDaoImpl. 2025-06-29 03:36:42,805 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceTagsDaoImpl. 2025-06-29 03:36:42,805 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ExternalLoadBalancerDeviceDaoImpl. 2025-06-29 03:36:42,805 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NicSecondaryIpDaoImpl. 2025-06-29 03:36:42,805 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectAccountJoinDaoImpl. 2025-06-29 03:36:42,805 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VGPUTypesDaoImpl. 2025-06-29 03:36:42,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AsyncJobJoinDaoImpl. 2025-06-29 03:36:42,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PortableIpRangeDaoImpl. 2025-06-29 03:36:42,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcDetailsDaoImpl. 2025-06-29 03:36:42,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainJoinDaoImpl. 2025-06-29 03:36:42,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkDomainDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RolePermissionsDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StorageNetworkIpRangeDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SecurityGroupVMMapDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NicDetailsDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostDetailsDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkOpDaoImpl. 2025-06-29 03:36:42,807 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConsoleSessionDaoImpl. 2025-06-29 03:36:42,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EngineHostPodDaoImpl. 2025-06-29 03:36:42,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterDrsPlanDaoImpl. 2025-06-29 03:36:42,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataCenterIpv4GuestSubnetDaoImpl. 2025-06-29 03:36:42,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackupRepositoryDaoImpl. 2025-06-29 03:36:42,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UploadDaoImpl. 2025-06-29 03:36:42,808 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KeystoreDaoImpl. 2025-06-29 03:36:42,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CertificateDaoImpl. 2025-06-29 03:36:42,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean GuestOSDaoImpl. 2025-06-29 03:36:42,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean OpRouterMonitorServiceDaoImpl. 2025-06-29 03:36:42,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageVMInstanceDaoImpl. 2025-06-29 03:36:42,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerNode. 2025-06-29 03:36:42,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DatabaseIntegrityChecker. 2025-06-29 03:36:42,809 INFO [c.c.u.DatabaseIntegrityChecker] (main:[]) (logid:) Grabbing lock to check for database integrity. 2025-06-29 03:36:42,810 INFO [c.c.u.DatabaseIntegrityChecker] (main:[]) (logid:) Performing database integrity check 2025-06-29 03:36:42,812 DEBUG [c.c.u.DatabaseIntegrityChecker] (main:[]) (logid:) No duplicate hosts with the same local storage found in database 2025-06-29 03:36:42,815 DEBUG [c.c.u.d.VersionDaoImpl] (main:[]) (logid:) Checking to see if the database is at a version before it was the version table is created 2025-06-29 03:36:42,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConfigurationServerImpl. 2025-06-29 03:36:42,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AsyncJobManagerImpl. 2025-06-29 03:36:42,827 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Cancel left-over job-3679 2025-06-29 03:36:42,829 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Unknown ResourceType. Skip Cleanup: Thread 2025-06-29 03:36:42,838 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Purge queue item for cancelled job-3679 2025-06-29 03:36:42,840 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Cancel left-over job-3680 2025-06-29 03:36:42,840 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Unknown ResourceType. Skip Cleanup: Thread 2025-06-29 03:36:42,842 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Purge queue item for cancelled job-3680 2025-06-29 03:36:42,843 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Cancel left-over job-3683 2025-06-29 03:36:42,843 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Unknown ResourceType. Skip Cleanup: Thread 2025-06-29 03:36:42,845 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:[]) (logid:) Purge queue item for cancelled job-3683 2025-06-29 03:36:42,862 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AsyncJobMonitor. 2025-06-29 03:36:42,862 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcManagerImpl. 2025-06-29 03:36:42,863 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QueryManagerImpl. 2025-06-29 03:36:42,863 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SharedFSServiceImpl. 2025-06-29 03:36:42,867 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TaggedResourceManagerImpl. 2025-06-29 03:36:42,868 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AnnotationManagerImpl. 2025-06-29 03:36:42,868 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectManagerImpl. 2025-06-29 03:36:42,869 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceCleanupServiceImpl. 2025-06-29 03:36:42,871 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VnfTemplateManagerImpl. 2025-06-29 03:36:42,871 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkModelImpl. 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. VpcVirtualRouter-VpcVirtualRouter-VpcVirtualRouterElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. InternalLbVm-InternalLbVm-InternalLoadBalancerElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. BaremetalDhcpProvider-BareMetalDhcp-BaremetalDhcpElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. BaremetalUserdataProvider-BareMetalUserdata-BaremetalUserdataElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. BaremetalPxeProvider-BareMetalPxe-BaremetalPxeElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. BigSwitchBcf-BigSwitchBcf-BigSwitchBcfElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. CiscoVnmc-CiscoVNMC-CiscoVnmcElement 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. JuniperContrailVpcRouter-ContrailVpcElement-ContrailVpcElementImpl 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. JuniperContrailRouter-ContrailElement-ContrailElementImpl 2025-06-29 03:36:42,872 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. GloboDns-GloboDnsElement-GloboDnsElement 2025-06-29 03:36:42,875 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. Netscaler-Netscaler-NetscalerElement 2025-06-29 03:36:42,875 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. Nsx-Nsx-NsxElement 2025-06-29 03:36:42,875 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. NiciraNvp-NiciraNvp-NiciraNvpElement 2025-06-29 03:36:42,875 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. Opendaylight-OpendaylightElement-OpendaylightElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. Ovs-Ovs-OvsElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. PaloAlto-PaloAlto-PaloAltoExternalFirewallElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. ConfigDrive-ConfigDrive-ConfigDriveNetworkElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. SecurityGroupProvider-SecurityGroupProvider-SecurityGroupElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. VirtualRouter-VirtualRouter-VirtualRouterElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. StratosphereSsp-StratosphereSsp-SspElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. Tungsten-TungstenElement-TungstenElement 2025-06-29 03:36:42,876 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Add provider <-> element map entry. BrocadeVcs-BrocadeVcs-BrocadeVcsElement 2025-06-29 03:36:42,891 INFO [c.c.n.NetworkModelImpl] (main:[]) (logid:) Started Network Model 2025-06-29 03:36:42,891 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApplicationLoadBalancerManagerImpl. 2025-06-29 03:36:42,891 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LoadBalancingRulesManagerImpl. 2025-06-29 03:36:42,891 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmManagerImpl. 2025-06-29 03:36:42,910 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserDataManagerImpl. 2025-06-29 03:36:42,910 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Ipv6AddressManagerImpl. 2025-06-29 03:36:42,910 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ExternalNetworkDeviceManagerImpl. 2025-06-29 03:36:42,910 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,911 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean IpAddressManagerImpl. 2025-06-29 03:36:42,911 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApiServer. 2025-06-29 03:36:42,930 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Global concurrent snapshot config parameter null is less or equal 0; defaulting to unlimited 2025-06-29 03:36:42,932 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Global concurrent migration config parameter 0 is less or equal 0; defaulting to unlimited 2025-06-29 03:36:42,932 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin UcsManagerImpl 2025-06-29 03:36:42,932 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ClusterDrsServiceImpl 2025-06-29 03:36:42,932 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin RoutedIpv4ManagerImpl 2025-06-29 03:36:42,932 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ApiDiscoveryServiceImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin APIAuthenticationManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin DiagnosticsServiceImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin CAManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin InternalLoadBalancerElement 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ResourceCleanupServiceImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin DirectDownloadManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin VnfTemplateManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin StorageBrowserImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin HAManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ProjectRoleManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ManagementServerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BaremetalVlanManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin Ipv6ServiceImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin VMScheduleManagerImpl 2025-06-29 03:36:42,935 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin VpcVirtualRouterElement 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BaremetalPxeManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin VmwareManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BackupManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BaremetalManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin AnnotationManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin VolumeImportUnmanageManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin SharedFSServiceImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin RoleManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin DedicatedResourceManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin CloudianConnectorImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin LdapManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin MetricsServiceImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin OAuth2AuthManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin QuotaServiceImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ApiRateLimitServiceImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin SAML2AuthManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ApiSolidFireIntegrationTestServiceImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ApiSiocServiceImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin KubernetesClusterManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin KubernetesVersionManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin UnmanagedVMsManagerImpl 2025-06-29 03:36:42,936 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BaremetalDhcpManagerImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BigSwitchBcfElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin CiscoVnmcElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ContrailVpcElementImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ContrailElementImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin GloboDnsElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin NetscalerElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin NsxElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin NsxProviderServiceImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin NiciraNvpElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin OpenDaylightControllerResourceManagerImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin OvsElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin PaloAltoExternalFirewallElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin VirtualRouterElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin SspElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin TungstenProviderServiceImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin BrocadeVcsElement 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin CiscoNexusVSMElement 2025-06-29 03:36:42,937 INFO [o.a.c.s.d.u.ElastistorVolumeApiServiceImpl] (main:[]) (logid:) Commands were registered successfully with elastistor volume api service. [cmdcount:3] 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin ElastistorVolumeApiServiceImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin WebhookApiServiceImpl 2025-06-29 03:36:42,937 DEBUG [c.c.a.ApiServer] (main:[]) (logid:) Discovered plugin WebhookServiceImpl 2025-06-29 03:36:42,989 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConsoleAccessManagerImpl. 2025-06-29 03:36:42,990 INFO [o.a.c.c.ConsoleAccessManagerImpl] (main:[]) (logid:) The ConsoleSessionCleanupTask will run every 180 hours 2025-06-29 03:36:42,991 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ReconcileCommandServiceImpl. 2025-06-29 03:36:42,991 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean OCFS2ManagerImpl. 2025-06-29 03:36:42,992 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HypervisorGuruManagerImpl. 2025-06-29 03:36:42,992 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLServiceImpl. 2025-06-29 03:36:42,992 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StorageOrchestrator. 2025-06-29 03:36:42,992 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Baremetal Manager. 2025-06-29 03:36:42,993 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UsageServiceImpl. 2025-06-29 03:36:42,993 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BaremetalKickStartServiceImpl. 2025-06-29 03:36:42,993 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:42,993 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TemplateManagerImpl. 2025-06-29 03:36:42,993 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HAManagerImpl. 2025-06-29 03:36:42,993 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkOrchestrator. 2025-06-29 03:36:42,995 INFO [o.a.c.e.o.NetworkOrchestrator] (main:[]) (logid:) Network Manager will run the NetworkGarbageCollector every '600' seconds. 2025-06-29 03:36:42,996 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean APIAuthenticationManagerImpl. 2025-06-29 03:36:42,997 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BucketApiServiceImpl. 2025-06-29 03:36:42,998 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmwareManagerImpl. 2025-06-29 03:36:42,998 DEBUG [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) checking to see if we should schedule a job to search for fully cloned templates to clean-up 2025-06-29 03:36:43,000 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) Cleanup mounted NFS mount points used in previous session 2025-06-29 03:36:43,000 INFO [c.c.h.v.m.VmwareManagerImpl] (main:[]) (logid:) start done 2025-06-29 03:36:43,000 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StorageBrowserImpl. 2025-06-29 03:36:43,000 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserPasswordResetManagerImpl. 2025-06-29 03:36:43,000 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DeploymentPlanningManagerImpl. 2025-06-29 03:36:43,004 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CapacityManagerImpl. 2025-06-29 03:36:43,006 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Baremetal Vlan Manager. 2025-06-29 03:36:43,013 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMLeaseManagerImpl. 2025-06-29 03:36:43,013 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkUsageManagerImpl. 2025-06-29 03:36:43,013 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Volume Manager. 2025-06-29 03:36:43,013 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceIconManagerImpl. 2025-06-29 03:36:43,013 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceManagerImpl. 2025-06-29 03:36:43,013 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotManagerImpl. 2025-06-29 03:36:43,016 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainManagerImpl. 2025-06-29 03:36:43,016 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RoleManagerImpl. 2025-06-29 03:36:43,016 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackgroundPollManagerImpl. 2025-06-29 03:36:43,017 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Scheduled background poll task: org.apache.cloudstack.ha.HAManagerImpl$HAManagerBgPollTask 2025-06-29 03:36:43,020 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Scheduled background poll task: org.apache.cloudstack.ca.CAManagerImpl$CABackgroundTask 2025-06-29 03:36:43,023 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Scheduled background poll task: org.apache.cloudstack.outofbandmanagement.OutOfBandManagementServiceImpl$OutOfBandManagementPowerStatePollTask 2025-06-29 03:36:43,025 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Scheduled background poll task: org.apache.cloudstack.backup.BackupManagerImpl$BackupSyncTask 2025-06-29 03:36:43,028 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:[]) (logid:) Scheduled background poll task: org.apache.cloudstack.diagnostics.DiagnosticsServiceImpl$GCBackgroundTask 2025-06-29 03:36:43,028 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StorageNetworkManagerImpl. 2025-06-29 03:36:43,028 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StatsCollector. 2025-06-29 03:36:43,031 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) register cluster listener class com.cloud.server.StatsCollector$ManagementServerStatusAdministrator 2025-06-29 03:36:43,046 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-d2fee096]) (logid:8db98c17) DbCollector is running... 2025-06-29 03:36:43,048 DEBUG [c.c.s.StatsCollector] (main:[]) (logid:) vm.disk.stats.interval - 0 is 0 or less than 0, so not scheduling the vm disk stats thread 2025-06-29 03:36:43,050 DEBUG [c.c.s.StatsCollector] (main:[]) (logid:) vm.network.stats.interval - 0 is 0 or less than 0, so not scheduling the vm network stats thread 2025-06-29 03:36:43,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterDrsServiceImpl. 2025-06-29 03:36:43,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PremiumSecondaryStorageManagerImpl. 2025-06-29 03:36:43,065 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (main:[]) (logid:) Start secondary storage vm manager 2025-06-29 03:36:43,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConfigurationManagerImpl. 2025-06-29 03:36:43,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusteredAgentManagerImpl. 2025-06-29 03:36:43,076 INFO [c.c.u.n.NioServer] (main:[]) (logid:) NioServer started and listening on /[0:0:0:0:0:0:0:0]:8250 2025-06-29 03:36:43,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (main:[]) (logid:) Scheduled direct agent scan task to run at an interval of 90000 seconds 2025-06-29 03:36:43,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RollingMaintenanceManagerImpl. 2025-06-29 03:36:43,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SnapshotSchedulerImpl. 2025-06-29 03:36:43,090 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BareMetalPingServiceImpl. 2025-06-29 03:36:43,090 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CAManagerImpl. 2025-06-29 03:36:43,093 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImageStoreUploadMonitorImpl. 2025-06-29 03:36:43,095 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VirtualNetworkApplianceManagerImpl. 2025-06-29 03:36:43,111 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DirectDownloadManagerImpl. 2025-06-29 03:36:43,113 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:43,116 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean InternalLoadBalancerVMManagerImpl. 2025-06-29 03:36:43,116 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterFenceManagerImpl. 2025-06-29 03:36:43,116 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceLimitManagerImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ResourceMetaDataManagerImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ConsoleProxyManagerImpl. 2025-06-29 03:36:43,120 INFO [c.c.c.ConsoleProxyManagerImpl] (main:[]) (logid:) Start console proxy manager 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcVirtualNetworkApplianceManagerImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackupRepositoryServiceImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BaremetalPxeManagerImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Site2SiteVpnManagerImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImageStoreServiceImpl. 2025-06-29 03:36:43,120 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DataStoreProviderManagerImpl. 2025-06-29 03:36:43,121 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean com.cloud.network.security.SecurityGroupManagerImpl2. 2025-06-29 03:36:43,139 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:43,140 DEBUG [o.a.c.o.OutOfBandManagementServiceImpl] (main:[]) (logid:) Discovered out-of-band management drivers configured in the OutOfBandManagementService 2025-06-29 03:36:43,140 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean null. 2025-06-29 03:36:43,140 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (main:[]) (logid:) LB HealthCheckmanager is getting Started 2025-06-29 03:36:43,141 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VolumeApiServiceImpl. 2025-06-29 03:36:43,141 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AccountManagerImpl. 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.ucs.manager.UcsManagerImpl 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.cluster.ClusterDrsServiceImpl 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.RoutedIpv4ManagerImpl 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.discovery.ApiDiscoveryServiceImpl 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.api.auth.APIAuthenticationManagerImpl 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.diagnostics.DiagnosticsServiceImpl 2025-06-29 03:36:43,141 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ca.CAManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.element.InternalLoadBalancerElement 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.resource.ResourceCleanupServiceImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.direct.download.DirectDownloadManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.template.VnfTemplateManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.browser.StorageBrowserImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ha.HAManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.acl.ProjectRoleManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.maintenance.ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.server.ManagementServerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.manager.BaremetalVlanManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.Ipv6ServiceImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.vm.schedule.VMScheduleManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.VpcVirtualRouterElement 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.networkservice.BaremetalPxeManagerImpl 2025-06-29 03:36:43,142 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.backup.BackupManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.manager.BaremetalManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.annotation.AnnotationManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.volume.VolumeImportUnmanageManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.sharedfs.SharedFSServiceImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.acl.RoleManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.dedicated.DedicatedResourceManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.cloudian.CloudianConnectorImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ldap.LdapManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.maintenance.ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.metrics.MetricsServiceImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.oauth2.OAuth2AuthManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.quota.QuotaServiceImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ratelimit.ApiRateLimitServiceImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.saml.SAML2AuthManagerImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.api.solidfire.ApiSolidFireIntegrationTestServiceImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.api.sioc.ApiSiocServiceImpl 2025-06-29 03:36:43,143 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.kubernetes.version.KubernetesVersionManagerImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.vm.UnmanagedVMsManagerImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.networkservice.BaremetalDhcpManagerImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.BigSwitchBcfElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.CiscoVnmcElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.contrail.management.ContrailElementImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.globo.globodns.cloudstack.element.GloboDnsElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.NetscalerElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.service.NsxElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.service.NsxProviderServiceImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.NiciraNvpElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.opendaylight.agent.OpenDaylightControllerResourceManagerImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.OvsElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.PaloAltoExternalFirewallElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.VirtualRouterElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.element.SspElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.tungsten.service.TungstenProviderServiceImpl 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.BrocadeVcsElement 2025-06-29 03:36:43,144 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.CiscoNexusVSMElement 2025-06-29 03:36:43,145 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.datastore.util.ElastistorVolumeApiServiceImpl 2025-06-29 03:36:43,145 INFO [o.a.c.s.d.u.ElastistorVolumeApiServiceImpl] (main:[]) (logid:) Commands were registered successfully with elastistor volume api service. [cmdcount:3] 2025-06-29 03:36:43,145 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.mom.webhook.WebhookApiServiceImpl 2025-06-29 03:36:43,145 DEBUG [c.c.u.AccountManagerImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.mom.webhook.WebhookServiceImpl 2025-06-29 03:36:43,148 INFO [c.c.u.AccountManagerImpl] (main:[]) (logid:) Api Discovery Service: Annotation, docstrings, api relation graph processed in 5.352958 ms 2025-06-29 03:36:43,150 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMSchedulerImpl. 2025-06-29 03:36:43,156 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMScheduleManagerImpl. 2025-06-29 03:36:43,157 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AutoScaleManagerImpl. 2025-06-29 03:36:43,159 INFO [c.c.n.a.AutoScaleManagerImpl] (main:[]) (logid:) AutoScale Manager created a thread pool to check autoscale vm groups. The pool size is : 10 2025-06-29 03:36:43,160 INFO [c.c.n.a.AutoScaleManagerImpl] (main:[]) (logid:) AutoScale Manager created a blocking queue to check autoscale vm groups. The queue size is : 10 2025-06-29 03:36:43,162 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkACLManagerImpl. 2025-06-29 03:36:43,163 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean managementServerMaintenanceManager. 2025-06-29 03:36:43,164 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AlertManagerImpl. 2025-06-29 03:36:43,165 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirewallManagerImpl. 2025-06-29 03:36:43,165 INFO [c.c.n.f.FirewallManagerImpl] (main:[]) (logid:) Firewall provider list is com.cloud.network.element.VpcVirtualRouterElement@3695e4f2 2025-06-29 03:36:43,165 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RulesManagerImpl. 2025-06-29 03:36:43,165 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NetworkServiceImpl. 2025-06-29 03:36:43,169 DEBUG [c.c.n.NetworkServiceImpl] (main:[]) (logid:) Discovered internal loadbalancer elements configured on NetworkServiceImpl 2025-06-29 03:36:43,169 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusteredVirtualMachineManagerImpl. 2025-06-29 03:36:43,182 INFO [c.c.v.ClusteredVirtualMachineManagerImpl] (main:[]) (logid:) Handling unfinished work item: ItWork[714e61b9-e23a-48cf-9567-a37fc2bdb613-Starting-303-Starting] 2025-06-29 03:36:43,190 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (main:[]) (logid:) HA schedule restart 2025-06-29 03:36:43,257 INFO [c.c.h.HighAvailabilityManagerExtImpl] (main:[]) (logid:) Schedule vm for HA: VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} 2025-06-29 03:36:43,257 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (main:[]) (logid:) Wakeup workers HA 2025-06-29 03:36:43,297 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DownloadMonitorImpl. 2025-06-29 03:36:43,297 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean EventDistributorImpl. 2025-06-29 03:36:43,297 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectRoleManagerImpl. 2025-06-29 03:36:43,298 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RemoteAccessVpnManagerImpl. 2025-06-29 03:36:43,298 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BackupManagerImpl. 2025-06-29 03:36:43,300 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DiagnosticsServiceImpl. 2025-06-29 03:36:43,301 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SyncQueueManagerImpl. 2025-06-29 03:36:43,301 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UploadMonitorImpl. 2025-06-29 03:36:43,302 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VMSnapshotManagerImpl. 2025-06-29 03:36:43,302 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HighAvailabilityManagerExtImpl. 2025-06-29 03:36:43,308 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-1:[]) (logid:) Starting work 2025-06-29 03:36:43,308 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-2:[]) (logid:) Starting work 2025-06-29 03:36:43,308 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[]) (logid:) Starting work 2025-06-29 03:36:43,309 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[]) (logid:) Starting work 2025-06-29 03:36:43,309 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-3:[]) (logid:) Starting work 2025-06-29 03:36:43,313 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KeystoreManagerImpl. 2025-06-29 03:36:43,313 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterManagerImpl. 2025-06-29 03:36:43,313 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Starting Cluster manager, msid: 90520738888109, mshost: null 2025-06-29 03:36:43,326 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Management server ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"}, runId 1751160968845 is being started 2025-06-29 03:36:43,337 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Management server (host : ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"}) is being started at 172.16.20.40:9090 2025-06-29 03:36:43,349 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) Cluster manager was started successfully 2025-06-29 03:36:43,350 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DumpRegistry. 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Vm Snapshot Strategies Registry] contains [StorageVMSnapshotStrategy, ScaleIOVMSnapshotStrategy, DefaultVMSnapshotStrategy, LinstorVMSnapshotStrategy, StorPoolVMSnapshotStrategy] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Network Elements Registry] contains [VpcVirtualRouter, InternalLbVm, BareMetalDhcp, BareMetalUserdata, BareMetalPxe, BigSwitchBcf, CiscoVNMC, ContrailVpcElement, ContrailElement, GloboDnsElement, Netscaler, Nsx, NiciraNvp, OpendaylightElement, Ovs, PaloAlto, ConfigDrive, SecurityGroupProvider, VirtualRouter, StratosphereSsp, TungstenElement, BrocadeVcs, CiscoNexus1000vVSM] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Static Nat Service Providers Registry] contains [VpcVirtualRouter, BigSwitchBcf, CiscoVNMC, ContrailVpcElement, ContrailElement, Netscaler, Nsx, NiciraNvp, Ovs, PaloAlto, VirtualRouter, TungstenElement] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Query Selectors Registry] contains [] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Network A C L Service Providers Registry] contains [VpcVirtualRouter, BigSwitchBcf, ContrailVpcElement, Nsx] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Vpc Providers Registry] contains [VpcVirtualRouter, ContrailVpcElement, Nsx] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Ha Fence Builders Registry] contains [OvmFenceBuilder, Ovm3FenceBuilder, KVMFenceBuilder, VMwareFenceBuilder, XenServerFenceBuilder] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Ha Investigators Registry] contains [SimpleInvestigator, XenServerInvestigator, KVMInvestigator, HypervInvestigator, VMwareInvestigator, PingInvestigator, ManagementIPSysVMInvestigator, Ovm3Investigator] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Pluggable A P I Authenticators Registry] contains [SAML2Auth, OAUTH2Auth] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Template Adapter Registry] contains [BareMetalAdapter, HypervisorAdapter] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Affinity Processors Registry] contains [HostAntiAffinityProcessor, ExplicitDedicationProcessor, HostAffinityProcessor, NonStrictHostAntiAffinityProcessor, NonStrictHostAffinityProcessor] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Kubernetes Service Helper Registry] contains [KubernetesServiceHelper] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Gslb Service Providers Registry] contains [Netscaler] 2025-06-29 03:36:43,351 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Ha Providers Registry] contains [KVMHAProvider] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Firewall Service Providers Registry] contains [VpcVirtualRouter, BigSwitchBcf, CiscoVNMC, Nsx, PaloAlto, VirtualRouter, TungstenElement] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Network Gurus Registry] contains [BigSwitchBcfGuestNetworkGuru, ContrailGuru, NsxPublicNetworkGuru, NsxGuestNetworkGuru, NiciraNvpGuestNetworkGuru, OpendaylightGuestNetworkGuru, OvsGuestNetworkGuru, ControlNetworkGuru, StorageNetworkGuru, DirectNetworkGuru, ExternalGuestNetworkGuru, DirectPodBasedNetworkGuru, PrivateNetworkGuru, PublicNetworkGuru, PodBasedNetworkGuru, SspGuestNetworkGuru, TungstenGuestNetworkGuru, BrocadeVcsGuestNetworkGuru, VxlanGuestNetworkGuru] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Host Allocators Registry] contains [FirstFitRouting] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Dhcp Providers Registry] contains [VpcVirtualRouter, BareMetalDhcp, ContrailVpcElement, ContrailElement, Nsx, ConfigDrive, VirtualRouter] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Snapshot Strategies Registry] contains [StorageSystemSnapshotStrategy, CephSnapshotStrategy, ScaleIOSnapshotStrategy, DefaultSnapshotStrategy, StorPoolSnapshotStrategy] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Deployment Planners Registry] contains [FirstFitPlanner, UserDispersingPlanner, UserConcentratedPodPlanner, ImplicitDedicationPlanner, BareMetalPlanner] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Ip Deployers Registry] contains [VpcVirtualRouter, BigSwitchBcf, CiscoVNMC, ContrailVpcElement, ContrailElement, Netscaler, Nsx, NiciraNvp, Ovs, PaloAlto, VirtualRouter, TungstenElement] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Port Forwarding Service Providers Registry] contains [VpcVirtualRouter, CiscoVNMC, Nsx, NiciraNvp, Ovs, PaloAlto, VirtualRouter, TungstenElement] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Site2 Site Vpn Service Providers Registry] contains [VpcVirtualRouter] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [User Two Factor Authenticators Registry] contains [totp, staticpin] 2025-06-29 03:36:43,352 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Event Bus Registry] contains [webhookEventBus] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Pod Allocators Registry] contains [UserFirst] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Storage Pool Allocators Registry] contains [LocalStorage, ClusterScopeStoragePoolAllocator, ZoneWideStoragePoolAllocator] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Resource Discoverers Registry] contains [Bare Metal Agent, Hyper-V Agent, Ovm Discover, Ovm3Discover, SecondaryStorage, KVM Agent, CustomHW Agent, Lxc Discover, VMware Discover, XCP Agent] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Data Motion Strategies Registry] contains [AncientDataMotionStrategy, StorageSystemDataMotionStrategy, HypervStorageMotionStrategy, KvmNonManagedStorageDataMotionStrategy, XenServerStorageMotionStrategy, StorPoolDataMotionStrategy, VmwareStorageMotionStrategy] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Security Checkers Registry] contains [AffinityGroupAccessChecker, DomainChecker] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Internal Load Balancer Element Service Registry] contains [InternalLbVm, Nsx] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Direct Download Registry] contains [] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Shared F S Providers Registry] contains [SHAREDFSVM] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Api Checkers Registry] contains [DynamicRoleBasedAPIAccessChecker, ProjectRoleBasedApiAccessChecker, StaticRoleBasedAPIAccessChecker, ApiRateLimitServiceImpl] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Load Balancing Service Providers Registry] contains [VpcVirtualRouter, InternalLbVm, Netscaler, Nsx, Ovs, VirtualRouter, TungstenElement] 2025-06-29 03:36:43,353 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Secondary Storage Vm Allocators Registry] contains [SecondaryStorageVmDefaultAllocator] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Hypervisor Gurus Registry] contains [BaremetalGuru, HypervGuru, OvmGuru, Ovm3Guru, KVMGuru, LXCGuru, VMwareGuru, XenServerGuru] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Backup Providers Registry] contains [dummy, nas, networker, veeam] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Out Of Band Management Drivers Registry] contains [IPMITOOL, NESTEDCLOUDSTACK, REDFISH] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [User Password Encoders Registry] contains [PBKDF2, SHA256SALT, SAML2, oauth2] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Ca Providers Registry] contains [RootCAProvider] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [DataStoreProviderManagerImpl] contains [CloudStackImageStoreProviderImpl, S3ImageStoreProviderImpl, SwiftImageStoreProviderImpl, CephObjectStoreProviderImpl, CloudianHyperStoreObjectStoreProviderImpl, MinIOObjectStoreProviderImpl, SimulatorObjectStoreProviderImpl, ElastistorPrimaryDataStoreProvider, DateraPrimaryDataStoreProvider, CloudStackPrimaryDataStoreProviderImpl, FlashArrayPrimaryDatastoreProviderImpl, LinstorPrimaryDatastoreProviderImpl, PrimeraPrimaryDatastoreProviderImpl, ScaleIOPrimaryDatastoreProvider, SolidFireSharedPrimaryDataStoreProvider, SolidFirePrimaryDataStoreProvider, StorPoolPrimaryDataStoreProvider] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Cluster Drs Algorithm Registry] contains [balanced, condensed] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Api Acl Checkers Registry] contains [DynamicRoleBasedAPIAccessChecker, ProjectRoleBasedApiAccessChecker, StaticRoleBasedAPIAccessChecker] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Remote Access V P N Service Provider Registry] contains [VpcVirtualRouter, VirtualRouter] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Ha Planners Registry] contains [SkipHeuresticsPlanner] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [Console Proxy Allocators Registry] contains [Balance] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.r.DumpRegistry] (main:[]) (logid:) Registry [User Authenticators Registry] contains [PBKDF2, SHA256SALT, MD5, LDAP, SAML2, oauth2] 2025-06-29 03:36:43,354 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmWorkJobDispatcher. 2025-06-29 03:36:43,355 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApiDiscoveryServiceImpl. 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.ucs.manager.UcsManagerImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.cluster.ClusterDrsServiceImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.RoutedIpv4ManagerImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.discovery.ApiDiscoveryServiceImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.api.auth.APIAuthenticationManagerImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.diagnostics.DiagnosticsServiceImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ca.CAManagerImpl 2025-06-29 03:36:43,355 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.element.InternalLoadBalancerElement 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.resource.ResourceCleanupServiceImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.direct.download.DirectDownloadManagerImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.template.VnfTemplateManagerImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.browser.StorageBrowserImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ha.HAManagerImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.acl.ProjectRoleManagerImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.maintenance.ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:43,356 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.server.ManagementServerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.manager.BaremetalVlanManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.Ipv6ServiceImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.vm.schedule.VMScheduleManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.VpcVirtualRouterElement 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.networkservice.BaremetalPxeManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.backup.BackupManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.manager.BaremetalManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.annotation.AnnotationManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.volume.VolumeImportUnmanageManagerImpl 2025-06-29 03:36:43,357 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.sharedfs.SharedFSServiceImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.acl.RoleManagerImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.dedicated.DedicatedResourceManagerImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.cloudian.CloudianConnectorImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ldap.LdapManagerImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.maintenance.ManagementServerMaintenanceManagerImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.metrics.MetricsServiceImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.oauth2.OAuth2AuthManagerImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.quota.QuotaServiceImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.ratelimit.ApiRateLimitServiceImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.saml.SAML2AuthManagerImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.api.solidfire.ApiSolidFireIntegrationTestServiceImpl 2025-06-29 03:36:43,358 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.api.sioc.ApiSiocServiceImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.kubernetes.version.KubernetesVersionManagerImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.vm.UnmanagedVMsManagerImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.baremetal.networkservice.BaremetalDhcpManagerImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.BigSwitchBcfElement 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.CiscoVnmcElement 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.contrail.management.ContrailVpcElementImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.contrail.management.ContrailElementImpl 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.globo.globodns.cloudstack.element.GloboDnsElement 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.NetscalerElement 2025-06-29 03:36:43,359 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.service.NsxElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.service.NsxProviderServiceImpl 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.NiciraNvpElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.opendaylight.agent.OpenDaylightControllerResourceManagerImpl 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.OvsElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.PaloAltoExternalFirewallElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.VirtualRouterElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.element.SspElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.network.tungsten.service.TungstenProviderServiceImpl 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.BrocadeVcsElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: com.cloud.network.element.CiscoNexusVSMElement 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.storage.datastore.util.ElastistorVolumeApiServiceImpl 2025-06-29 03:36:43,360 INFO [o.a.c.s.d.u.ElastistorVolumeApiServiceImpl] (main:[]) (logid:) Commands were registered successfully with elastistor volume api service. [cmdcount:3] 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.mom.webhook.WebhookApiServiceImpl 2025-06-29 03:36:43,360 DEBUG [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) getting api commands of service: org.apache.cloudstack.mom.webhook.WebhookServiceImpl 2025-06-29 03:36:43,776 INFO [o.a.c.d.ApiDiscoveryServiceImpl] (main:[]) (logid:) Api Discovery Service: Annotation, docstrings, api relation graph processed in 421.249079 ms 2025-06-29 03:36:43,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VpcVirtualRouter. 2025-06-29 03:36:43,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RoutedIpv4ManagerImpl. 2025-06-29 03:36:43,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApiAsyncJobDispatcher. 2025-06-29 03:36:43,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean VmWorkJobWakeupDispatcher. 2025-06-29 03:36:43,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean InternalLbVm. 2025-06-29 03:36:43,777 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Ipv6ServiceImpl. 2025-06-29 03:36:43,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterBasedAgentLoadBalancerPlanner. 2025-06-29 03:36:43,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CloudZonesStartupProcessor. 2025-06-29 03:36:43,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean IndirectAgentLBServiceImpl. 2025-06-29 03:36:43,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean cloud-init. 2025-06-29 03:36:43,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BASIC. 2025-06-29 03:36:43,778 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ClusterServiceServletAdapter. 2025-06-29 03:36:43,794 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Basic. 2025-06-29 03:36:43,795 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ManagementServerImpl. 2025-06-29 03:36:43,795 INFO [c.c.s.ManagementServerImpl] (main:[]) (logid:) Startup CloudStack management server... 2025-06-29 03:36:43,795 INFO [c.c.c.C.ListenerThread] (Thread-24:[]) (logid:) Cluster service servlet container listening on host: 172.16.20.40 and port 9090 2025-06-29 03:36:43,797 INFO [c.c.c.ClusterManagerImpl] (main:[]) (logid:) register cluster listener class com.cloud.server.LockControllerListener 2025-06-29 03:36:43,799 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,800 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [allocator] context. 2025-06-29 03:36:43,800 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [allocator]. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,800 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [host-allocator-random] context. 2025-06-29 03:36:43,800 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [host-allocator-random]. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RandomAllocator. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,800 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [planner] context. 2025-06-29 03:36:43,800 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [planner]. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,800 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [api-planner] context. 2025-06-29 03:36:43,800 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [api-planner]. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserDispersingPlanner. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,800 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-planner] context. 2025-06-29 03:36:43,800 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [baremetal-planner]. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BareMetalPlanner. 2025-06-29 03:36:43,800 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,800 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [explicit-dedication] context. 2025-06-29 03:36:43,801 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [explicit-dedication]. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ExplicitDedicationProcessor. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,801 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [host-affinity] context. 2025-06-29 03:36:43,801 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [host-affinity]. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostAffinityProcessor. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,801 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [host-anti-affinity] context. 2025-06-29 03:36:43,801 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [host-anti-affinity]. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HostAntiAffinityProcessor. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,801 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [implicit-dedication] context. 2025-06-29 03:36:43,801 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [implicit-dedication]. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ImplicitDedicationPlanner. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,801 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [non-strict-host-affinity] context. 2025-06-29 03:36:43,801 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [non-strict-host-affinity]. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NonStrictHostAffinityProcessor. 2025-06-29 03:36:43,801 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,801 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [non-strict-host-anti-affinity] context. 2025-06-29 03:36:43,802 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [non-strict-host-anti-affinity]. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean NonStrictHostAntiAffinityProcessor. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,802 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-planner] context. 2025-06-29 03:36:43,802 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [server-planner]. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirstFitPlanner. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,802 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [skip-heurestics] context. 2025-06-29 03:36:43,802 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [skip-heurestics]. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SkipHeuresticsPlanner. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,802 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [user-concentrated-pod] context. 2025-06-29 03:36:43,802 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [user-concentrated-pod]. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserConcentratedPodPlanner. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,802 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-allocator] context. 2025-06-29 03:36:43,802 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [server-allocator]. 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,802 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RecreateHostAllocator. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirstFitAllocator. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean Balance. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean FirstFitRouting. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean TestingAllocator. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,803 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [api] context. 2025-06-29 03:36:43,803 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [api]. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,803 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [acl-dynamic-role-based] context. 2025-06-29 03:36:43,803 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [acl-dynamic-role-based]. 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DynamicRoleBasedAPIAccessChecker. 2025-06-29 03:36:43,805 INFO [o.a.c.s.d.u.ElastistorVolumeApiServiceImpl] (main:[]) (logid:) Commands were registered successfully with elastistor volume api service. [cmdcount:3] 2025-06-29 03:36:43,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,806 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [acl-project-role-based] context. 2025-06-29 03:36:43,806 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [acl-project-role-based]. 2025-06-29 03:36:43,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ProjectRoleBasedApiAccessChecker. 2025-06-29 03:36:43,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,806 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [acl-static-role-based] context. 2025-06-29 03:36:43,806 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [acl-static-role-based]. 2025-06-29 03:36:43,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,806 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean StaticRoleBasedAPIAccessChecker. 2025-06-29 03:36:43,809 INFO [o.a.c.s.d.u.ElastistorVolumeApiServiceImpl] (main:[]) (logid:) Commands were registered successfully with elastistor volume api service. [cmdcount:3] 2025-06-29 03:36:43,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,809 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [cloudian] context. 2025-06-29 03:36:43,809 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [cloudian]. 2025-06-29 03:36:43,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean CloudianConnectorImpl. 2025-06-29 03:36:43,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,809 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ldap] context. 2025-06-29 03:36:43,809 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [ldap]. 2025-06-29 03:36:43,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,809 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LdapConfigurationDaoImpl. 2025-06-29 03:36:43,810 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LdapTrustMapDaoImpl. 2025-06-29 03:36:43,810 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LDAP. 2025-06-29 03:36:43,810 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean LdapManagerImpl. 2025-06-29 03:36:43,810 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,810 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [maintenance] context. 2025-06-29 03:36:43,810 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [maintenance]. 2025-06-29 03:36:43,810 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,810 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean managementServerMaintenanceManager. 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,812 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [md5] context. 2025-06-29 03:36:43,812 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [md5]. 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean MD5. 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,812 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [metrics] context. 2025-06-29 03:36:43,812 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [metrics]. 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean MetricsServiceImpl. 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,812 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [oauth2] context. 2025-06-29 03:36:43,812 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [oauth2]. 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,812 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean OauthProviderDaoImpl. 2025-06-29 03:36:43,813 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean OAUTH2Auth. 2025-06-29 03:36:43,814 INFO [o.a.c.o.OAuth2AuthManagerImpl] (main:[]) (logid:) OAUTH plugin not enabled so not loading 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean oauth2. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean github. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean google. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,815 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [pbkdf2] context. 2025-06-29 03:36:43,815 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [pbkdf2]. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PBKDF2. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,815 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [plaintext] context. 2025-06-29 03:36:43,815 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [plaintext]. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PLAINTEXT. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,815 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [prometheus] context. 2025-06-29 03:36:43,815 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [prometheus]. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PrometheusExporterImpl. 2025-06-29 03:36:43,815 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean PrometheusExporterServerImpl. 2025-06-29 03:36:43,816 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,817 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [quota] context. 2025-06-29 03:36:43,817 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [quota]. 2025-06-29 03:36:43,817 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,817 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaTariffDaoImpl. 2025-06-29 03:36:43,817 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaEmailConfigurationDaoImpl. 2025-06-29 03:36:43,817 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaAccountDaoImpl. 2025-06-29 03:36:43,817 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaBalanceDaoImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaCreditsDaoImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaEmailTemplatesDaoImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaUsageDaoImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean UserVmDetailsDaoImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaServiceImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaAlertManagerImpl. 2025-06-29 03:36:43,818 INFO [o.a.c.q.QuotaAlertManagerImpl] (main:[]) (logid:) Starting Alert Manager 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaStatementImpl. 2025-06-29 03:36:43,819 INFO [o.a.c.q.QuotaStatementImpl] (main:[]) (logid:) Starting Statement Manager 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean QuotaManagerImpl. 2025-06-29 03:36:43,819 INFO [o.a.c.q.QuotaManagerImpl] (main:[]) (logid:) Starting Quota Manager 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,819 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [rate-limit] context. 2025-06-29 03:36:43,819 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [rate-limit]. 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApiRateLimitServiceImpl. 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,819 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [saml2] context. 2025-06-29 03:36:43,819 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [saml2]. 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SAMLTokenDaoImpl. 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SAML2. 2025-06-29 03:36:43,819 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SAML2Auth. 2025-06-29 03:36:43,820 INFO [o.a.c.s.SAML2AuthManagerImpl] (main:[]) (logid:) SAML auth plugin not enabled so not loading 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,820 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [server-api] context. 2025-06-29 03:36:43,820 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [server-api]. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean DomainChecker. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean AffinityGroupAccessChecker. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,820 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [sha256salted] context. 2025-06-29 03:36:43,820 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [sha256salted]. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean SHA256SALT. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,820 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [solidfire-intg-test] context. 2025-06-29 03:36:43,820 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [solidfire-intg-test]. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApiSolidFireIntegrationTestServiceImpl. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,820 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [staticpin] context. 2025-06-29 03:36:43,820 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [staticpin]. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean staticpin. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,820 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [totp] context. 2025-06-29 03:36:43,820 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [totp]. 2025-06-29 03:36:43,820 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean totp. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,821 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [vmware-sioc] context. 2025-06-29 03:36:43,821 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [vmware-sioc]. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean ApiSiocServiceImpl. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,821 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [api-config] context. 2025-06-29 03:36:43,821 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [api-config]. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,821 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [backend] context. 2025-06-29 03:36:43,821 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [backend]. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,821 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [backup] context. 2025-06-29 03:36:43,821 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [backup]. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,821 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [dummy-backup] context. 2025-06-29 03:36:43,821 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [dummy-backup]. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean dummy. 2025-06-29 03:36:43,821 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,821 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [nas] context. 2025-06-29 03:36:43,822 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [nas]. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean nas. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,822 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [networker] context. 2025-06-29 03:36:43,822 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [networker]. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean networker. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,822 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [veeam] context. 2025-06-29 03:36:43,822 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [veeam]. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean veeam. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,822 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [ca] context. 2025-06-29 03:36:43,822 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [ca]. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,822 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [root-ca] context. 2025-06-29 03:36:43,822 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [root-ca]. 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,822 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean RootCAProvider. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,882 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [compute] context. 2025-06-29 03:36:43,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [compute]. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,882 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [baremetal-compute] context. 2025-06-29 03:36:43,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [baremetal-compute]. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean BaremetalGuru. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,882 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [hyperv-compute] context. 2025-06-29 03:36:43,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [hyperv-compute]. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HypervInvestigator. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean HypervGuru. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,882 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [kubernetes] context. 2025-06-29 03:36:43,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [kubernetes]. 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,882 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:43,882 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [kubernetes-service] context. 2025-06-29 03:36:43,882 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Starting module [kubernetes-service]. 2025-06-29 03:36:43,883 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Starting CloudStack Components 2025-06-29 03:36:43,883 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KubernetesSupportedVersionDaoImpl. 2025-06-29 03:36:43,883 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KubernetesClusterDaoImpl. 2025-06-29 03:36:43,883 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KubernetesClusterVmMapDaoImpl. 2025-06-29 03:36:43,884 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KubernetesClusterDetailsDaoImpl. 2025-06-29 03:36:43,884 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KubernetesVersionManagerImpl. 2025-06-29 03:36:43,884 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) starting bean KubernetesClusterManagerImpl. 2025-06-29 03:36:44,014 DEBUG [c.c.k.c.KubernetesClusterManagerImpl] (main:[ctx-03ab8d1b]) (logid:) Found default role for Kubernetes service account in projects: Role {"id":9,"name":"Project Kubernetes Service Role","roleType":"User","uuid":"26c68c8f-898e-49e4-87c7-a678191fc94e"} 2025-06-29 03:36:44,021 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KubernetesServiceHelper. 2025-06-29 03:36:44,030 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,030 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [kvm-compute] context. 2025-06-29 03:36:44,030 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [kvm-compute]. 2025-06-29 03:36:44,030 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,030 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KVMInvestigator. 2025-06-29 03:36:44,030 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KVMHostActivityChecker. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KVMHAProvider. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,031 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ovm-compute] context. 2025-06-29 03:36:44,031 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ovm-compute]. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvmFenceBuilder. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvmGuru. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,031 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ovm3-compute] context. 2025-06-29 03:36:44,031 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ovm3-compute]. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Ovm3Investigator. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Ovm3FenceBuilder. 2025-06-29 03:36:44,031 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Ovm3Guru. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,032 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-alert-adapter-compute] context. 2025-06-29 03:36:44,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-alert-adapter-compute]. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ConsoleProxyAlertAdapter. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,032 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-compute] context. 2025-06-29 03:36:44,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-compute]. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KVMGuru. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean LXCGuru. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,032 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-fencer] context. 2025-06-29 03:36:44,032 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-fencer]. 2025-06-29 03:36:44,032 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean RecreatableFencer. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KVMFenceBuilder. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,033 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-investigator] context. 2025-06-29 03:36:44,033 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-investigator]. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean XenServerInvestigator. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ManagementIPSysVMInvestigator. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SimpleInvestigator. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean PingInvestigator. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,033 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [vmware-compute] context. 2025-06-29 03:36:44,033 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [vmware-compute]. 2025-06-29 03:36:44,033 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VMwareFenceBuilder. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VMwareGuru. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VMwareInvestigator. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,034 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [xenserver-compute] context. 2025-06-29 03:36:44,034 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [xenserver-compute]. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean XenServerGuru. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean XenServerFenceBuilder. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,034 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [network] context. 2025-06-29 03:36:44,034 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [network]. 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,034 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,034 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [baremetal-network] context. 2025-06-29 03:36:44,035 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [baremetal-network]. 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean null. 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BareMetalDhcp. 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BareMetalUserdata. 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BareMetalPxe. 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,035 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [bigswitch] context. 2025-06-29 03:36:44,035 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [bigswitch]. 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,035 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BigSwitchBcfDaoImpl. 2025-06-29 03:36:44,036 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BigSwitchBcf. 2025-06-29 03:36:44,036 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BigSwitchBcfGuestNetworkGuru. 2025-06-29 03:36:44,036 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,036 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [cisco-vnmc] context. 2025-06-29 03:36:44,036 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [cisco-vnmc]. 2025-06-29 03:36:44,036 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,036 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean CiscoVnmcDaoImpl. 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NetworkAsa1000vMapDaoImpl. 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean CiscoAsa1000vDaoImpl. 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean CiscoVNMC. 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,037 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [contrail] context. 2025-06-29 03:36:44,037 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [contrail]. 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ContrailManagerImpl. 2025-06-29 03:36:44,037 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ContrailGuru. 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ContrailVpcElement. 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ContrailElement. 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,038 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [elb] context. 2025-06-29 03:36:44,038 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [elb]. 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ElasticLbVmMapDaoImpl. 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ElasticLoadBalancerManagerImpl. 2025-06-29 03:36:44,038 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,038 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [globodns] context. 2025-06-29 03:36:44,039 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [globodns]. 2025-06-29 03:36:44,039 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,039 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean GloboDnsElement. 2025-06-29 03:36:44,039 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,039 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [netscaler] context. 2025-06-29 03:36:44,039 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [netscaler]. 2025-06-29 03:36:44,039 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,039 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NetScalerPodDaoImpl. 2025-06-29 03:36:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) ManagementServerCollector is running... 2025-06-29 03:36:44,040 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NetScalerServicePackageDaoImpl. 2025-06-29 03:36:44,040 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NetScalerControlCenterDaoImpl. 2025-06-29 03:36:44,040 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NetScalerVMManagerImpl. 2025-06-29 03:36:44,040 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Netscaler. 2025-06-29 03:36:44,040 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,041 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [nsx] context. 2025-06-29 03:36:44,041 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [nsx]. 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Nsx. 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NsxPublicNetworkGuru. 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NsxGuestNetworkGuru. 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,041 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [nvp] context. 2025-06-29 03:36:44,041 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [nvp]. 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,041 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NiciraNvpDaoImpl. 2025-06-29 03:36:44,042 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NiciraNvpNicMappingDaoImpl. 2025-06-29 03:36:44,042 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NiciraNvpRouterMappingDaoImpl. 2025-06-29 03:36:44,042 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NiciraNvpGuestNetworkGuru. 2025-06-29 03:36:44,042 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NiciraNvp. 2025-06-29 03:36:44,042 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,042 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [opendaylight] context. 2025-06-29 03:36:44,043 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [opendaylight]. 2025-06-29 03:36:44,043 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,043 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OpenDaylightControllerMappingDaoImpl. 2025-06-29 03:36:44,043 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OpendaylightGuestNetworkGuru. 2025-06-29 03:36:44,043 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OpendaylightElement. 2025-06-29 03:36:44,043 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,043 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ovs] context. 2025-06-29 03:36:44,043 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ovs]. 2025-06-29 03:36:44,043 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,044 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvsTunnelNetworkDaoImpl. 2025-06-29 03:36:44,044 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VpcDistributedRouterSeqNoDaoImpl. 2025-06-29 03:36:44,044 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvsTunnelInterfaceDaoImpl. 2025-06-29 03:36:44,044 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvsNetworkTopologyGuruImpl. 2025-06-29 03:36:44,044 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvsTunnelManagerImpl. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean OvsGuestNetworkGuru. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Ovs. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,045 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [paloalto] context. 2025-06-29 03:36:44,045 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [paloalto]. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean PaloAlto. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,045 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-network] context. 2025-06-29 03:36:44,045 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-network]. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ControlNetworkGuru. 2025-06-29 03:36:44,045 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ConfigDrive. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean StorageNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean DirectNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ExternalGuestNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean DirectPodBasedNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SecurityGroupProvider. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VirtualRouter. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean PrivateNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean PublicNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean PodBasedNetworkGuru. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,046 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ssp] context. 2025-06-29 03:36:44,046 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ssp]. 2025-06-29 03:36:44,046 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,047 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SspUuidDaoImpl. 2025-06-29 03:36:44,047 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) DB properties were already loaded 2025-06-29 03:36:44,047 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SspTenantDaoImpl. 2025-06-29 03:36:44,047 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:36:44,047 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SspCredentialDaoImpl. 2025-06-29 03:36:44,047 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean StratosphereSsp. 2025-06-29 03:36:44,047 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SspGuestNetworkGuru. 2025-06-29 03:36:44,047 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,047 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [tungsten] context. 2025-06-29 03:36:44,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [tungsten]. 2025-06-29 03:36:44,048 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,048 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean TungstenFabricLBHealthMonitorDaoImpl. 2025-06-29 03:36:44,048 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean TungstenProviderServiceImpl. 2025-06-29 03:36:44,048 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean TungstenServiceImpl. 2025-06-29 03:36:44,057 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean TungstenGuestNetworkGuru. 2025-06-29 03:36:44,057 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean TungstenElement. 2025-06-29 03:36:44,057 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,058 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [vcs] context. 2025-06-29 03:36:44,058 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [vcs]. 2025-06-29 03:36:44,058 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,058 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BrocadeVcsDaoImpl. 2025-06-29 03:36:44,058 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BrocadeVcsNetworkVlanMappingDaoImpl. 2025-06-29 03:36:44,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BrocadeVcs. 2025-06-29 03:36:44,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BrocadeVcsGuestNetworkGuru. 2025-06-29 03:36:44,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,059 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [vmware-network] context. 2025-06-29 03:36:44,059 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [vmware-network]. 2025-06-29 03:36:44,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean CiscoNexus1000vVSM. 2025-06-29 03:36:44,059 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,059 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [vxlan] context. 2025-06-29 03:36:44,060 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [vxlan]. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VxlanGuestNetworkGuru. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,060 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-alert-adapter-backend] context. 2025-06-29 03:36:44,060 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-alert-adapter-backend]. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ClusterAlertAdapter. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,060 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage] context. 2025-06-29 03:36:44,060 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage]. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean StorageVMSnapshotStrategy. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ScaleIOVMSnapshotStrategy. 2025-06-29 03:36:44,060 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean DefaultVMSnapshotStrategy. 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,061 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [baremetal-storage] context. 2025-06-29 03:36:44,061 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [baremetal-storage]. 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean BareMetalAdapter. 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,061 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-alert-adapter-storage] context. 2025-06-29 03:36:44,061 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-alert-adapter-storage]. 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SecondaryStorageVmAlertAdapter. 2025-06-29 03:36:44,061 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,061 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-storage] context. 2025-06-29 03:36:44,061 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-storage]. 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SecondaryStorageVmDefaultAllocator. 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,062 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-template-adapter] context. 2025-06-29 03:36:44,062 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-template-adapter]. 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean HypervisorAdapter. 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,062 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [sharedfs-provider-storagevm] context. 2025-06-29 03:36:44,062 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [sharedfs-provider-storagevm]. 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SHAREDFSVM. 2025-06-29 03:36:44,062 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,062 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-allocator] context. 2025-06-29 03:36:44,062 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-allocator]. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean GCStorage. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean UserFirst. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ZoneWideStoragePoolAllocator. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean LocalStorage. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean ClusterScopeStoragePoolAllocator. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,063 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-image-default] context. 2025-06-29 03:36:44,063 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-image-default]. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,063 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-image-s3] context. 2025-06-29 03:36:44,063 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-image-s3]. 2025-06-29 03:36:44,063 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,064 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-image-swift] context. 2025-06-29 03:36:44,064 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-image-swift]. 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,064 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-object-ceph] context. 2025-06-29 03:36:44,064 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-object-ceph]. 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,064 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-object-cloudian] context. 2025-06-29 03:36:44,064 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-object-cloudian]. 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,064 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,064 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-object-minio] context. 2025-06-29 03:36:44,065 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-object-minio]. 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,065 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-object-simulator] context. 2025-06-29 03:36:44,065 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-object-simulator]. 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,065 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-cloudbyte] context. 2025-06-29 03:36:44,065 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-cloudbyte]. 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean elastistorVolumeApiService. 2025-06-29 03:36:44,065 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,065 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-datera] context. 2025-06-29 03:36:44,066 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-datera]. 2025-06-29 03:36:44,066 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,066 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,066 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-default] context. 2025-06-29 03:36:44,066 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Execution of process [4064722] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:36:44,066 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-default]. 2025-06-29 03:36:44,066 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Exit value of process [4064722] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:36:44,066 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,066 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-flasharray] context. 2025-06-29 03:36:44,066 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-flasharray]. 2025-06-29 03:36:44,066 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Process [4064722] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:36:44,066 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,066 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) usage install: null 2025-06-29 03:36:44,066 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,066 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-linstor] context. 2025-06-29 03:36:44,067 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-linstor]. 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean LinstorVMSnapshotStrategy. 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,067 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Sessions active: 0 2025-06-29 03:36:44,067 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-primera] context. 2025-06-29 03:36:44,067 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-primera]. 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,067 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-scaleio] context. 2025-06-29 03:36:44,067 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-scaleio]. 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,067 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,067 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-solidfire] context. 2025-06-29 03:36:44,067 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-solidfire]. 2025-06-29 03:36:44,068 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,068 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,068 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [storage-volume-storpool] context. 2025-06-29 03:36:44,068 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [storage-volume-storpool]. 2025-06-29 03:36:44,068 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,068 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean StorPoolAbandonObjectsCollector. 2025-06-29 03:36:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:36:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064726] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:36:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:36:44,080 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean StorPoolStatsCollector. 2025-06-29 03:36:44,082 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean StorPoolVMSnapshotStrategy. 2025-06-29 03:36:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064733] for command [/bin/bash -c uname -r ]. 2025-06-29 03:36:44,082 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,082 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [vmware-storage] context. 2025-06-29 03:36:44,082 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [vmware-storage]. 2025-06-29 03:36:44,082 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,082 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:36:44,082 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [cluster] context. 2025-06-29 03:36:44,083 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [cluster]. 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,083 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [balanced] context. 2025-06-29 03:36:44,083 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [balanced]. 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean balanced. 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,083 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [condensed] context. 2025-06-29 03:36:44,083 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [condensed]. 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean condensed. 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,083 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [discoverer] context. 2025-06-29 03:36:44,083 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [discoverer]. 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,083 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,083 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [baremetal-discoverer] context. 2025-06-29 03:36:44,083 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [baremetal-discoverer]. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Bare Metal Agent. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,084 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [hyperv-discoverer] context. 2025-06-29 03:36:44,084 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [hyperv-discoverer]. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Hyper-V Agent. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,084 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ovm-discoverer] context. 2025-06-29 03:36:44,084 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ovm-discoverer]. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Ovm Discover. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,084 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ovm3-discoverer] context. 2025-06-29 03:36:44,084 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ovm3-discoverer]. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Ovm3Discover. 2025-06-29 03:36:44,084 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,084 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [secondary-storage-discoverer] context. 2025-06-29 03:36:44,085 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [secondary-storage-discoverer]. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean SecondaryStorage. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,085 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [server-discoverer] context. 2025-06-29 03:36:44,085 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [server-discoverer]. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean KVM Agent. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean dummyHostDiscoverer. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean CustomHW Agent. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean Lxc Discover. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,085 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [vmware-discoverer] context. 2025-06-29 03:36:44,085 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [vmware-discoverer]. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean VMware Discover. 2025-06-29 03:36:44,085 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,085 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [xenserver-discoverer] context. 2025-06-29 03:36:44,085 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [xenserver-discoverer]. 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean XCP Agent. 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,086 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [event] context. 2025-06-29 03:36:44,086 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [event]. 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,086 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [webhook] context. 2025-06-29 03:36:44,086 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [webhook]. 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,086 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean WebhookDeliveryDaoImpl. 2025-06-29 03:36:44,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean WebhookDeliveryJoinDaoImpl. 2025-06-29 03:36:44,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean WebhookDaoImpl. 2025-06-29 03:36:44,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean WebhookJoinDaoImpl. 2025-06-29 03:36:44,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean webhookEventBus. 2025-06-29 03:36:44,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean WebhookApiServiceImpl. 2025-06-29 03:36:44,087 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean WebhookServiceImpl. 2025-06-29 03:36:44,091 DEBUG [o.a.c.m.w.WebhookServiceImpl] (main:[ctx-03ab8d1b]) (logid:) Scheduling webhook deliveries cleanup task with initial delay=180s and interval=3600s 2025-06-29 03:36:44,092 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,092 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [outofbandmanagement] context. 2025-06-29 03:36:44,092 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [outofbandmanagement]. 2025-06-29 03:36:44,092 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,092 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,092 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [ipmitool] context. 2025-06-29 03:36:44,092 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [ipmitool]. 2025-06-29 03:36:44,093 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,093 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean IPMITOOL. 2025-06-29 03:36:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064736] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:36:44,095 DEBUG [o.a.c.u.p.ProcessRunner] (main:[ctx-03ab8d1b]) (logid:) Preparing command [/usr/bin/ipmitool -V] to execute. 2025-06-29 03:36:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) system memory from /proc: 4887515136 2025-06-29 03:36:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:36:44,101 DEBUG [o.a.c.u.p.ProcessRunner] (main:[ctx-03ab8d1b]) (logid:) Submitting command [/usr/bin/ipmitool -V]. 2025-06-29 03:36:44,102 DEBUG [o.a.c.u.p.ProcessRunner] (main:[ctx-03ab8d1b]) (logid:) Waiting for a response from command [/usr/bin/ipmitool -V]. Defined timeout: [300]. 2025-06-29 03:36:44,104 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064744] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:36:44,104 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) free memory from /proc: 210071552 2025-06-29 03:36:44,105 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:36:44,106 DEBUG [o.a.c.u.p.ProcessRunner] (main:[ctx-03ab8d1b]) (logid:) Process standard output for command [/usr/bin/ipmitool -V]: [ipmitool version 1.8.18-csv ]. 2025-06-29 03:36:44,106 DEBUG [o.a.c.u.p.ProcessRunner] (main:[ctx-03ab8d1b]) (logid:) Process standard error output command [/usr/bin/ipmitool -V]: []. 2025-06-29 03:36:44,107 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (main:[ctx-03ab8d1b]) (logid:) OutOfBandManagementDriver ipmitool initialized [ipmitool version 1.8.18-csv ]. 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,107 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [nested-cloudstack] context. 2025-06-29 03:36:44,107 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [nested-cloudstack]. 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean NESTEDCLOUDSTACK. 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,107 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Trying to obtain module [redfish] context. 2025-06-29 03:36:44,107 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[ctx-03ab8d1b]) (logid:) Starting module [redfish]. 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Starting CloudStack Components 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) starting bean REDFISH. 2025-06-29 03:36:44,107 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Starting CloudStack Components 2025-06-29 03:36:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064752] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:36:44,129 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) used memory from /proc: 1045148 2025-06-29 03:36:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:36:44,139 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064754] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:36:44,140 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:36:44,164 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064756] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:36:44,165 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:36:44,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064758] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:36:44,177 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:36:44,184 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064765] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:36:44,184 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:36:44,193 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Successfully executed process [4064766] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:36:44,207 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail buffersdirect.capacity, 27031 2025-06-29 03:36:44,208 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail buffersdirect.count, 5 2025-06-29 03:36:44,208 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail buffersdirect.used, 27031 2025-06-29 03:36:44,209 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail buffersmapped.capacity, 0 2025-06-29 03:36:44,209 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail buffersmapped.count, 0 2025-06-29 03:36:44,209 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail buffersmapped.used, 0 2025-06-29 03:36:44,209 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:36:44,209 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:36:44,209 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail gcPS-Scavenge.count, 103 2025-06-29 03:36:44,210 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail gcPS-Scavenge.time, 427 2025-06-29 03:36:44,210 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:36:44,210 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail jvmuptime, 63913 2025-06-29 03:36:44,210 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:36:44,210 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memoryheap.committed, 480772096 2025-06-29 03:36:44,210 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memoryheap.init, 77594624 2025-06-29 03:36:44,211 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memoryheap.usage, 0.19936847974886707 2025-06-29 03:36:44,211 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorynon-heap.committed, 176947200 2025-06-29 03:36:44,212 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:36:44,212 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorynon-heap.max, -1 2025-06-29 03:36:44,212 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorynon-heap.usage, -1.75182096E8 2025-06-29 03:36:44,212 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorynon-heap.used, 175186880 2025-06-29 03:36:44,212 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.29710526315789476 2025-06-29 03:36:44,212 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.06555855938416422 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.18517762522078182 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.Compressed-Class-Space.usage, 0.011629633605480194 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.Metaspace.usage, 0.9954595962562657 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.PS-Eden-Space.usage, 0.32814803123474123 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.PS-Old-Gen.usage, 0.10723952939771375 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorypools.PS-Survivor-Space.usage, 1.0 2025-06-29 03:36:44,213 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorytotal.committed, 657719296 2025-06-29 03:36:44,214 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorytotal.init, 85262336 2025-06-29 03:36:44,214 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorytotal.max, 1908932607 2025-06-29 03:36:44,214 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail memorytotal.used, 555771416 2025-06-29 03:36:44,217 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail threadsdeadlocks, [] 2025-06-29 03:36:44,218 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail threadsnew.count, 0 2025-06-29 03:36:44,220 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Configuring CloudStack Components 2025-06-29 03:36:44,220 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[ctx-03ab8d1b]) (logid:) Done Configuring CloudStack Components 2025-06-29 03:36:44,222 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) not storing detail threadstimed_waiting.count, 84 2025-06-29 03:36:44,229 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-34f97b9e]) (logid:62d4074e) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":480772096,"freeJvmMemoryBytes":115167040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":63770,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.44,"totalInit":85262336,"totalUsed":540374248,"totalCommitted":657326080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":[],"agentCount":0,"heapMemoryUsed":380580992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":7,"threadsTerminatedCount":0,"threadsTotalCount":175,"threadsWaitingCount":84,"systemMemoryTotal":4887515136,"systemMemoryFree":210071552,"systemMemoryUsed":1045148,"systemMemoryVirtualSize":6187687936,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.44,0.58,0.29],"systemCyclesUsage":[66392439,53819578,4588645785],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:36:44,235 INFO [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Setup cluster service servlet. service url: https://172.16.20.40:9090/clusterservice, request timeout: 300 seconds 2025-06-29 03:36:44,255 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 1, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":480772096,"freeJvmMemoryBytes":115167040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":63770,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.44,"totalInit":85262336,"totalUsed":540374248,"totalCommitted":657326080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":[],"agentCount":0,"heapMemoryUsed":380580992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":7,"threadsTerminatedCount":0,"threadsTotalCount":175,"threadsWaitingCount":84,"systemMemoryTotal":4887515136,"systemMemoryFree":210071552,"systemMemoryUsed":1045148,"systemMemoryVirtualSize":6187687936,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.44,0.58,0.29],"systemCyclesUsage":[66392439,53819578,4588645785],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:36:44,255 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:36:44,282 INFO [c.c.u.LogUtils] (main:[ctx-03ab8d1b]) (logid:) log4j configuration found at /etc/cloudstack/management/log4j-cloud.xml 2025-06-29 03:36:44,503 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-7:[ctx-a451f4ef]) (logid:3081475a) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":480772096,"freeJvmMemoryBytes":115167040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":63770,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.44,"totalInit":85262336,"totalUsed":540374248,"totalCommitted":657326080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":[],"agentCount":0,"heapMemoryUsed":380580992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":7,"threadsTerminatedCount":0,"threadsTotalCount":175,"threadsWaitingCount":84,"systemMemoryTotal":4887515136,"systemMemoryFree":210071552,"systemMemoryUsed":1045148,"systemMemoryVirtualSize":6187687936,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.44,0.58,0.29],"systemCyclesUsage":[66392439,53819578,4588645785],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:36:44,532 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 134 ms 2025-06-29 03:36:44,542 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 277ms. agent: 0, pdu seq: 1, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":480772096,"freeJvmMemoryBytes":115167040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":63770,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.44,"totalInit":85262336,"totalUsed":540374248,"totalCommitted":657326080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":[],"agentCount":0,"heapMemoryUsed":380580992,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":7,"threadsTerminatedCount":0,"threadsTotalCount":175,"threadsWaitingCount":84,"systemMemoryTotal":4887515136,"systemMemoryFree":210071552,"systemMemoryUsed":1045148,"systemMemoryVirtualSize":6187687936,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.44,0.58,0.29],"systemCyclesUsage":[66392439,53819578,4588645785],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:36:44,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-002e9d37]) (logid:42110d1d) We are good, no orphan management server msid in host table is found 2025-06-29 03:36:44,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-002e9d37]) (logid:42110d1d) No inactive management server node found 2025-06-29 03:36:44,879 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-002e9d37]) (logid:42110d1d) Detected management node joined, ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"}, nodeIP:172.16.20.40 2025-06-29 03:36:44,884 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-002e9d37]) (logid:42110d1d) No inactive management server node found 2025-06-29 03:36:44,885 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-002e9d37]) (logid:42110d1d) Peer scan is finished. profiler: Done. Duration: 10ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 2ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 3ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:45,390 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-8-thread-1:[]) (logid:) A client/agent attempting connection from address=10.25.0.2 has presented these certificate(s): Certificate [1] : Serial: c1a76f67b5605a4f Not Before:Tue Sep 17 02:13:58 CEST 2024 Not After:Wed Sep 17 14:13:58 CEST 2025 Signature Algorithm:SHA256withRSA Version:3 Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=cs-compute1.cloud.redacted.net Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:[[7, 10.25.0.2], [2, cs-compute1.cloud.redacted.net], [2, 10.25.0.2]] Certificate [2] : Serial: 6163766eca31086f Not Before:Sat Sep 07 02:28:00 CEST 2024 Not After:Mon Aug 31 14:28:00 CEST 2054 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=ca.cloudstack.apache.org Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:null 2025-06-29 03:36:45,409 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-8-thread-1:[]) (logid:) Client/agent connection from ip=10.25.0.2 has been validated and trusted. 2025-06-29 03:36:45,453 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-9-thread-1:[]) (logid:) A client/agent attempting connection from address=10.25.0.3 has presented these certificate(s): Certificate [1] : Serial: d9264c7f4a55e25e Not Before:Tue Sep 17 02:16:25 CEST 2024 Not After:Wed Sep 17 14:16:25 CEST 2025 Signature Algorithm:SHA256withRSA Version:3 Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=cs-compute2.cloud.redacted.net Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:[[7, 10.25.0.3], [2, cs-compute2.cloud.redacted.net], [2, 10.25.0.3]] Certificate [2] : Serial: 6163766eca31086f Not Before:Sat Sep 07 02:28:00 CEST 2024 Not After:Mon Aug 31 14:28:00 CEST 2054 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=ca.cloudstack.apache.org Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:null 2025-06-29 03:36:45,456 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-9-thread-1:[]) (logid:) Client/agent connection from ip=10.25.0.3 has been validated and trusted. 2025-06-29 03:36:45,617 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-10-thread-1:[]) (logid:) A client/agent attempting connection from address=10.25.0.191 has presented these certificate(s): Certificate [1] : Serial: 6a18020fdaf82e96 Not Before:Fri Jun 20 01:59:07 CEST 2025 Not After:Sat Jun 20 13:59:07 CEST 2026 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=s-2-VM Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:[[7, 169.254.180.250], [7, 10.25.0.191], [7, 10.26.0.11], [2, s-2-VM]] Certificate [2] : Serial: 6163766eca31086f Not Before:Sat Sep 07 02:28:00 CEST 2024 Not After:Mon Aug 31 14:28:00 CEST 2054 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=ca.cloudstack.apache.org Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:null 2025-06-29 03:36:45,621 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-10-thread-1:[]) (logid:) Client/agent connection from ip=10.25.0.191 has been validated and trusted. 2025-06-29 03:36:45,731 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-11-thread-1:[]) (logid:) A client/agent attempting connection from address=10.25.0.132 has presented these certificate(s): Certificate [1] : Serial: eb9c6e148a25c0dd Not Before:Fri Jun 27 21:00:55 CEST 2025 Not After:Sun Jun 28 09:00:55 CEST 2026 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=v-306-VM Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:[[7, 169.254.96.226], [7, 10.25.0.132], [7, 10.26.0.170], [2, v-306-VM]] Certificate [2] : Serial: 6163766eca31086f Not Before:Sat Sep 07 02:28:00 CEST 2024 Not After:Mon Aug 31 14:28:00 CEST 2054 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=ca.cloudstack.apache.org Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:null 2025-06-29 03:36:45,733 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-11-thread-1:[]) (logid:) Client/agent connection from ip=10.25.0.132 has been validated and trusted. 2025-06-29 03:36:45,752 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:[]) (logid:) Seq 4-12268: Scheduling the first command { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 101, [{"com.cloud.agent.api.StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-2-VM-PremiumSecondaryStorageResource","name":"s-2-VM","id":"4","version":"4.20.1.0","iqn":"NoIqn","publicIpAddress":"10.26.0.11","publicNetmask":"255.255.255.0","publicMacAddress":"1e:00:44:00:00:67","privateIpAddress":"10.25.0.191","privateMacAddress":"1e:00:40:00:00:5c","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.191","storageNetmask":"255.255.255.0","storageMacAddress":"1e:00:40:00:00:5c","resourceName":"PremiumSecondaryStorageResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,758 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-12268: Processing the first command { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 101, [{"com.cloud.agent.api.StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-2-VM-PremiumSecondaryStorageResource","name":"s-2-VM","id":"4","version":"4.20.1.0","iqn":"NoIqn","publicIpAddress":"10.26.0.11","publicNetmask":"255.255.255.0","publicMacAddress":"1e:00:44:00:00:67","privateIpAddress":"10.25.0.191","privateMacAddress":"1e:00:40:00:00:5c","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.191","storageNetmask":"255.255.255.0","storageMacAddress":"1e:00:40:00:00:5c","resourceName":"PremiumSecondaryStorageResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,773 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to CiscoVnmcElement 2025-06-29 03:36:45,774 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement 2025-06-29 03:36:45,774 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer 2025-06-29 03:36:45,774 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl 2025-06-29 03:36:45,774 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer 2025-06-29 03:36:45,774 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp 2025-06-29 03:36:45,774 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer 2025-06-29 03:36:45,775 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) createHostVOForConnectedAgent: Host {"id":4,"name":"s-2-VM","type":"SecondaryStorageVM","uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:36:45,775 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl 2025-06-29 03:36:45,811 DEBUG [c.c.h.d.HostDaoImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Resource state update: [id = 4; name = s-2-VM; old state = Enabled; event = InternalCreated; new state = Enabled] 2025-06-29 03:36:45,811 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) [Resource state = Enabled, Agent event = , Host = AgentConnected] 2025-06-29 03:36:45,827 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 7-8243: Scheduling the first command { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 101, [{"com.cloud.agent.api.StartupProxyCommand":{"proxyPort":"80","proxyVmId":"306","type":"ConsoleProxy","dataCenter":"1","pod":"1","guid":"Proxy.306-ConsoleProxyResource","name":"v-306-VM","id":"7","version":"4.20.1.0","iqn":"NoIqn","publicIpAddress":"10.26.0.170","publicNetmask":"255.255.255.0","publicMacAddress":"1e:00:75:00:01:06","privateIpAddress":"10.25.0.132","privateMacAddress":"1e:00:47:00:00:21","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.132","storageNetmask":"255.255.255.0","storageMacAddress":"1e:00:47:00:00:21","resourceName":"ConsoleProxyResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,833 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-8243: Processing the first command { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 101, [{"com.cloud.agent.api.StartupProxyCommand":{"proxyPort":"80","proxyVmId":"306","type":"ConsoleProxy","dataCenter":"1","pod":"1","guid":"Proxy.306-ConsoleProxyResource","name":"v-306-VM","id":"7","version":"4.20.1.0","iqn":"NoIqn","publicIpAddress":"10.26.0.170","publicNetmask":"255.255.255.0","publicMacAddress":"1e:00:75:00:01:06","privateIpAddress":"10.25.0.132","privateMacAddress":"1e:00:47:00:00:21","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.132","storageNetmask":"255.255.255.0","storageMacAddress":"1e:00:47:00:00:21","resourceName":"ConsoleProxyResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,844 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to CiscoVnmcElement 2025-06-29 03:36:45,844 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement 2025-06-29 03:36:45,844 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer 2025-06-29 03:36:45,844 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl 2025-06-29 03:36:45,844 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer 2025-06-29 03:36:45,844 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp 2025-06-29 03:36:45,845 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer 2025-06-29 03:36:45,846 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) createHostVOForConnectedAgent: Host {"id":7,"name":"v-306-VM","type":"ConsoleProxy","uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:36:45,846 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl 2025-06-29 03:36:45,844 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) create ClusteredAgentAttache for Host {"id":4,"name":"s-2-VM","type":"SecondaryStorageVM","uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:36:45,846 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovs 2025-06-29 03:36:45,846 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to VmwareServerDiscoverer 2025-06-29 03:36:45,846 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NsxElement 2025-06-29 03:36:45,846 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl 2025-06-29 03:36:45,852 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: XcpServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,852 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,852 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: HypervServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,853 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,853 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: TungstenElement, for rebalance: false 2025-06-29 03:36:45,853 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,855 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: NsxElement, for rebalance: false 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: ClusteredVirtualMachineManagerImpl, for rebalance: false 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: SecurityGroupListener, for rebalance: false 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,856 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Received a host startup notification 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: SecondaryStorageListener, for rebalance: false 2025-06-29 03:36:45,856 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,857 INFO [c.c.s.s.SecondaryStorageListener] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Received a host startup notification com.cloud.agent.api.StartupSecondaryStorageCommand 2025-06-29 03:36:45,861 DEBUG [c.c.h.d.HostDaoImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Resource state update: [id = 7; name = v-306-VM; old state = Enabled; event = InternalCreated; new state = Enabled] 2025-06-29 03:36:45,861 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) [Resource state = Enabled, Agent event = , Host = AgentConnected] 2025-06-29 03:36:45,875 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) create ClusteredAgentAttache for Host {"id":7,"name":"v-306-VM","type":"ConsoleProxy","uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:36:45,879 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: XcpServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,879 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,879 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: HypervServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,879 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,880 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: TungstenElement, for rebalance: false 2025-06-29 03:36:45,880 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,881 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: NsxElement, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: ClusteredVirtualMachineManagerImpl, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: SecurityGroupListener, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Received a host startup notification 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: SecondaryStorageListener, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: DeploymentPlanningManagerImpl, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: VmwareManagerImpl, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: NetworkOrchestrator, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: StoragePoolMonitor, for rebalance: false 2025-06-29 03:36:45,882 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: LocalStoragePoolListener, for rebalance: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: StorageCapacityListener, for rebalance: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: ComputeCapacityListener, for rebalance: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: DirectNetworkStatsListener, for rebalance: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: BehindOnPingListener, for rebalance: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: SetHostParamsListener, for rebalance: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: VirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,887 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: ConsoleProxyListener, for rebalance: false 2025-06-29 03:36:45,887 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,891 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.SecStorageSetupCommand is 1800 seconds 2025-06-29 03:36:45,899 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 7-3938116399158788097: Routed from 90520738888109 2025-06-29 03:36:45,902 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788097: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://nas.redacted.net/volume1/cs-secondary","_role":"Image","nfsVersion":"4"}},"secUrl":"nfs://nas.redacted.net/volume1/cs-secondary","postUploadKey":"xak2YNBuh2VNhpp4ZG-dxieydz8yW4Jj5mmBFFRkqI344MKlIqKWkVRHNJQniyFSi0YGSLxhgMe2agnQ1V_q3A","nfsVersion":"4","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,902 DEBUG [c.c.c.C.VmBasedAgentHook] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) SSL is disabled for console proxy. To enable SSL, please configure consoleproxy.sslEnabled and consoleproxy.url.domain global settings. 2025-06-29 03:36:45,918 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Wait time setting on com.cloud.agent.api.proxy.StartConsoleProxyAgentHttpHandlerCommand is 1800 seconds 2025-06-29 03:36:45,921 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704513: Routed from 90520738888109 2025-06-29 03:36:45,923 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704513: Sending { Cmd , MgmtId: 90520738888109, via: 7(v-306-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.proxy.StartConsoleProxyAgentHttpHandlerCommand":{"isSourceIpCheckEnabled":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,979 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 7-443886038272704513: Processing: { Ans: , MgmtId: 90520738888109, via: 7, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:45,979 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704513: Received: { Ans: , MgmtId: 90520738888109, via: 7(v-306-VM), Ver: v1, Flags: 110, { Answer } } 2025-06-29 03:36:45,980 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:[]) (logid:) Seq 7-443886038272704513: No more commands found 2025-06-29 03:36:45,980 INFO [c.c.c.C.VmBasedAgentHook] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Successfully sent out command to start HTTP handling in console proxy agent 2025-06-29 03:36:45,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:45,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: VpcVirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:45,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: DownloadListener, for rebalance: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: UploadListener, for rebalance: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: KvmServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: CustomServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Sending Connect to listener: LxcServerDiscoverer, for rebalance: false 2025-06-29 03:36:45,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) process connection to issue: null for host: 7, forRebalance: false, connection transferred: false 2025-06-29 03:36:45,987 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Wait time setting on com.cloud.agent.api.ReadyCommand is 60 seconds 2025-06-29 03:36:45,989 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704514: Routed from 90520738888109 2025-06-29 03:36:45,991 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704514: Sending { Cmd , MgmtId: 90520738888109, via: 7(v-306-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"7","hostUuid":"fc16264e-ee6a-403f-809d-20f1822cb15a","hostName":"v-306-VM","enableHumanReadableSizes":"true","wait":"60","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,098 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 7-443886038272704514: Processing: { Ans: , MgmtId: 90520738888109, via: 7, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,098 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704514: Received: { Ans: , MgmtId: 90520738888109, via: 7(v-306-VM), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:46,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) [Resource state = Enabled, Agent event = , Host = Ready] 2025-06-29 03:36:46,099 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:[]) (logid:) Seq 7-443886038272704514: No more commands found 2025-06-29 03:36:46,114 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Wait time setting on com.cloud.agent.api.ReadyCommand is 1800 seconds 2025-06-29 03:36:46,116 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704515: Routed from 90520738888109 2025-06-29 03:36:46,117 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704515: Sending { Cmd , MgmtId: 90520738888109, via: 7(v-306-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"7","hostUuid":"fc16264e-ee6a-403f-809d-20f1822cb15a","hostName":"v-306-VM","enableHumanReadableSizes":"true","arch":"x86_64","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,236 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 7-443886038272704515: Processing: { Ans: , MgmtId: 90520738888109, via: 7, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,236 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:[]) (logid:) Seq 7-443886038272704515: No more commands found 2025-06-29 03:36:46,236 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-2:[ctx-ea2e3429]) (logid:d223c127) Seq 7-443886038272704515: Received: { Ans: , MgmtId: 90520738888109, via: 7(v-306-VM), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:46,350 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) Notify management server node join to listeners. 2025-06-29 03:36:46,351 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) Joining node, IP: 172.16.20.40, ms: ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:36:46,352 DEBUG [c.c.a.ClusterAlertAdapter] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) Receive cluster alert, EventArgs: com.cloud.cluster.ClusterNodeJoinEventArgs 2025-06-29 03:36:46,353 DEBUG [c.c.a.ClusterAlertAdapter] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) Handle cluster node join alert, joined node: 172.16.20.40 (ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"}) 2025-06-29 03:36:46,353 DEBUG [c.c.a.ClusterAlertAdapter] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) Management server node 172.16.20.40 (ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"}) is up, send alert 2025-06-29 03:36:46,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8768ee7]) (logid:66b10e99) No inactive management server node found 2025-06-29 03:36:46,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8768ee7]) (logid:66b10e99) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:46,365 WARN [c.c.a.AlertManagerImpl] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) alertType=[14] dataCenterId=[0] podId=[null] clusterId=[null] message=[Management server node 172.16.20.40 is up]. 2025-06-29 03:36:46,386 WARN [c.c.a.AlertManagerImpl] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Management server node 172.16.20.40 is up] and content []. 2025-06-29 03:36:46,388 WARN [c.c.c.ClusterManagerImpl$5] (Cluster-Notification-1:[ctx-b8504aac]) (logid:d962ea45) Notifying management server join event took 37 ms 2025-06-29 03:36:46,561 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[]) (logid:) Seq 4-3938116399158788097: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.SecStorageSetupAnswer":{"_dir":"f62efded-3f07-3561-a9c9-2429b1e27783","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,562 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:[]) (logid:) Seq 7-3938116399158788097: No more commands found 2025-06-29 03:36:46,562 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788097: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 110, { SecStorageSetupAnswer } } 2025-06-29 03:36:46,562 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success 2025-06-29 03:36:46,575 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Successfully programmed secondary storage [nas1] in secondary storage VM [s-2-VM]. 2025-06-29 03:36:46,581 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 1-741: Scheduling the first command { Cmd , MgmtId: -1, via: 1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"1","cpus":"16","speed":"4463","cpuArch":"x86_64","memory":"65549033472","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"4.18.0-553.58.1.el8_10.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"8.10","host.volume.encryption":"true","host.instance.conversion":"false","secured":"true","Host.OS":"Red Hat Enterprise Linux"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"6a680d56-20e5-39e2-b48d-240eb5d552cc-LibvirtComputingResource","name":"cs-compute1.cloud.redacted.net","id":"1","version":"4.21.0.0-SNAPSHOT","iqn":"iqn.1994-05.com.redhat:ce7749e17b2b","publicIpAddress":"10.25.0.2","publicNetmask":"255.255.255.0","publicMacAddress":"50:eb:f6:ef:e9:b4","privateIpAddress":"10.25.0.2","privateMacAddress":"50:eb:f6:ef:e9:b4","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.2","storageNetmask":"255.255.255.0","storageMacAddress":"50:eb:f6:ef:e9:b4","resourceName":"LibvirtComputingResource","gatewayIpAddress":"10.25.0.1","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa","host":"10.25.0.2","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(850.65 GB) 913383583744","availableBytes":"(455.18 GB) 488744812544"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"6a680d56-20e5-39e2-b48d-240eb5d552cc-LibvirtComputingResource","name":"cs-compute1.cloud.redacted.net","id":"1","version":"4.21.0.0-SNAPSHOT","resourceName":"LibvirtComputingResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,585 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.SecStorageFirewallCfgCommand is 1800 seconds 2025-06-29 03:36:46,587 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-741: Processing the first command { Cmd , MgmtId: -1, via: 1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"1","cpus":"16","speed":"4463","cpuArch":"x86_64","memory":"65549033472","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"4.18.0-553.58.1.el8_10.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"8.10","host.volume.encryption":"true","host.instance.conversion":"false","secured":"true","Host.OS":"Red Hat Enterprise Linux"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"6a680d56-20e5-39e2-b48d-240eb5d552cc-LibvirtComputingResource","name":"cs-compute1.cloud.redacted.net","id":"1","version":"4.21.0.0-SNAPSHOT","iqn":"iqn.1994-05.com.redhat:ce7749e17b2b","publicIpAddress":"10.25.0.2","publicNetmask":"255.255.255.0","publicMacAddress":"50:eb:f6:ef:e9:b4","privateIpAddress":"10.25.0.2","privateMacAddress":"50:eb:f6:ef:e9:b4","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.2","storageNetmask":"255.255.255.0","storageMacAddress":"50:eb:f6:ef:e9:b4","resourceName":"LibvirtComputingResource","gatewayIpAddress":"10.25.0.1","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa","host":"10.25.0.2","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(850.65 GB) 913383583744","availableBytes":"(455.18 GB) 488744812544"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"6a680d56-20e5-39e2-b48d-240eb5d552cc-LibvirtComputingResource","name":"cs-compute1.cloud.redacted.net","id":"1","version":"4.21.0.0-SNAPSHOT","resourceName":"LibvirtComputingResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,588 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 7-3938116399158788098: Routed from 90520738888109 2025-06-29 03:36:46,589 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788098: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.SecStorageFirewallCfgCommand":{"isAppendAIp":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,593 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to CiscoVnmcElement 2025-06-29 03:36:46,593 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement 2025-06-29 03:36:46,593 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer 2025-06-29 03:36:46,593 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl 2025-06-29 03:36:46,593 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer 2025-06-29 03:36:46,594 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp 2025-06-29 03:36:46,594 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer 2025-06-29 03:36:46,594 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) createHostVOForConnectedAgent: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:46,595 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl 2025-06-29 03:36:46,595 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovs 2025-06-29 03:36:46,595 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to VmwareServerDiscoverer 2025-06-29 03:36:46,595 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NsxElement 2025-06-29 03:36:46,595 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl 2025-06-29 03:36:46,595 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer 2025-06-29 03:36:46,656 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 4-3938116399158788098: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,657 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788098: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { Answer } } 2025-06-29 03:36:46,657 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Successfully created firewall rules into secondary storage VM [s-2-VM]. 2025-06-29 03:36:46,667 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.SecStorageVMSetupCommand is 1800 seconds 2025-06-29 03:36:46,671 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 7-3938116399158788099: Routed from 90520738888109 2025-06-29 03:36:46,672 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788099: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SecStorageVMSetupCommand":{"allowedInternalSites":[],"copyUserName":"cloud","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,681 DEBUG [c.c.h.d.HostDaoImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Resource state update: [id = 1; name = cs-compute1.cloud.redacted.net; old state = Enabled; event = InternalCreated; new state = Enabled] 2025-06-29 03:36:46,681 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) [Resource state = Enabled, Agent event = , Host = AgentConnected] 2025-06-29 03:36:46,695 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) create ClusteredAgentAttache for Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:46,699 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: XcpServerDiscoverer, for rebalance: false 2025-06-29 03:36:46,699 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,699 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Not XenServer so moving on. 2025-06-29 03:36:46,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: HypervServerDiscoverer, for rebalance: false 2025-06-29 03:36:46,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,700 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Not Hyper-V hypervisor, so moving on. 2025-06-29 03:36:46,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: TungstenElement, for rebalance: false 2025-06-29 03:36:46,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,701 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,702 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: NsxElement, for rebalance: false 2025-06-29 03:36:46,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: ClusteredVirtualMachineManagerImpl, for rebalance: false 2025-06-29 03:36:46,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,703 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Received startup command from hypervisor host. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:46,704 INFO [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Reset VM power state sync for host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:46,739 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,739 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: SecurityGroupListener, for rebalance: false 2025-06-29 03:36:46,739 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,739 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Received a host startup notification 2025-06-29 03:36:46,741 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203585: Routed from 90520738888109 2025-06-29 03:36:46,743 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203585: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":"1824","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,744 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Scheduled network rules cleanup, interval=1824 2025-06-29 03:36:46,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,744 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Received a host startup notification 2025-06-29 03:36:46,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: SecondaryStorageListener, for rebalance: false 2025-06-29 03:36:46,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: DeploymentPlanningManagerImpl, for rebalance: false 2025-06-29 03:36:46,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,747 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,747 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: VmwareManagerImpl, for rebalance: false 2025-06-29 03:36:46,747 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,747 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,747 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: NetworkOrchestrator, for rebalance: false 2025-06-29 03:36:46,747 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,751 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Host's hypervisorType is: KVM 2025-06-29 03:36:46,764 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending CheckNetworkCommand to check the Network is setup correctly on Agent 2025-06-29 03:36:46,766 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Wait time setting on com.cloud.agent.api.CheckNetworkCommand is 1800 seconds 2025-06-29 03:36:46,768 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203586: Routed from 90520738888109 2025-06-29 03:36:46,770 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203586: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.CheckNetworkCommand":{"networkInfoList":[{"physicalNetworkId":"200","privateNetworkName":"cloudbr0","publicNetworkName":"cloudbr0","guestNetworkName":"cloudbr0","storageNetworkName":"cloudbr0"}],"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,773 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:[]) (logid:) Seq 1-4725683383995203585: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 4-3938116399158788099: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,805 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:[]) (logid:) Seq 1-3938116399158788099: No more commands found 2025-06-29 03:36:46,805 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788099: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 110, { Answer } } 2025-06-29 03:36:46,805 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Details from executing class com.cloud.agent.api.SecStorageVMSetupCommand: 2025-06-29 03:36:46,805 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Successfully set HTTP auth into secondary storage VM [s-2-VM]. 2025-06-29 03:36:46,805 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: DeploymentPlanningManagerImpl, for rebalance: false 2025-06-29 03:36:46,805 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: VmwareManagerImpl, for rebalance: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: NetworkOrchestrator, for rebalance: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: StoragePoolMonitor, for rebalance: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: LocalStoragePoolListener, for rebalance: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: StorageCapacityListener, for rebalance: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: ComputeCapacityListener, for rebalance: false 2025-06-29 03:36:46,806 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: DirectNetworkStatsListener, for rebalance: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: BehindOnPingListener, for rebalance: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: SetHostParamsListener, for rebalance: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: VirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:46,807 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: ConsoleProxyListener, for rebalance: false 2025-06-29 03:36:46,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:46,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: VpcVirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:46,810 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,813 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: DownloadListener, for rebalance: false 2025-06-29 03:36:46,813 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,817 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:[]) (logid:) Seq 1-4725683383995203586: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.CheckNetworkAnswer":{"_reconnect":"false","result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,818 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:[]) (logid:) Seq 1-4725683383995203586: No more commands found 2025-06-29 03:36:46,819 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203586: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 110, { CheckNetworkAnswer } } 2025-06-29 03:36:46,819 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Network setup is correct on Agent 2025-06-29 03:36:46,819 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,820 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: StoragePoolMonitor, for rebalance: false 2025-06-29 03:36:46,820 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:46,845 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) getCommandHostDelegation: class com.cloud.agent.api.storage.ListVolumeCommand 2025-06-29 03:36:46,845 DEBUG [c.c.h.XenServerGuru] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:46,845 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.storage.ListVolumeCommand is 1800 seconds 2025-06-29 03:36:46,845 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} connected, connecting host to shared pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} and sending storage pool information ... 2025-06-29 03:36:46,849 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 1-3938116399158788100: Routed from 90520738888109 2025-06-29 03:36:46,851 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788100: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.ListVolumeCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://nas.redacted.net/volume1/cs-secondary","_role":"Image","nfsVersion":"4"}},"secUrl":"nfs://nas.redacted.net/volume1/cs-secondary","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,855 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Adding pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} to host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:46,906 DEBUG [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending modify storage pool command to agent: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} for storage pool: StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} with timeout 300 seconds 2025-06-29 03:36:46,908 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 2-734: Scheduling the first command { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"1","cpus":"8","speed":"2400","cpuArch":"x86_64","memory":"15326699520","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"4.18.0-553.16.1.el8_10.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"8.10","host.volume.encryption":"true","host.instance.conversion":"false","secured":"true","Host.OS":"Red Hat Enterprise Linux"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"42055e9b-5ec7-3a35-9818-606ee1da20ea-LibvirtComputingResource","name":"cs-compute2.cloud.redacted.net","id":"2","version":"4.20.1.0","iqn":"iqn.1994-05.com.redhat:ce7749e17b2b","publicIpAddress":"10.25.0.3","publicNetmask":"255.255.255.0","publicMacAddress":"00:23:24:54:e2:1a","privateIpAddress":"10.25.0.3","privateMacAddress":"00:23:24:54:e2:1a","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.3","storageNetmask":"255.255.255.0","storageMacAddress":"00:23:24:54:e2:1a","resourceName":"LibvirtComputingResource","gatewayIpAddress":"10.25.0.1","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d","host":"10.25.0.3","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(153.39 GB) 164705353728","availableBytes":"(46.80 GB) 50255364096"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"42055e9b-5ec7-3a35-9818-606ee1da20ea-LibvirtComputingResource","name":"cs-compute2.cloud.redacted.net","id":"2","version":"4.20.1.0","resourceName":"LibvirtComputingResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,911 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Wait time setting on com.cloud.agent.api.ModifyStoragePoolCommand is 300 seconds 2025-06-29 03:36:46,914 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203587: Routed from 90520738888109 2025-06-29 03:36:46,916 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203587: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"4","uuid":"10805e0a-2205-3158-993f-9350b08b9137","host":"nas.redacted.net","path":"/volume1/cs-primary","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//10805e0a-2205-3158-993f-9350b08b9137","details":{},"wait":"300","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,917 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-734: Processing the first command { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"1","cpus":"8","speed":"2400","cpuArch":"x86_64","memory":"15326699520","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"4.18.0-553.16.1.el8_10.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"8.10","host.volume.encryption":"true","host.instance.conversion":"false","secured":"true","Host.OS":"Red Hat Enterprise Linux"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"42055e9b-5ec7-3a35-9818-606ee1da20ea-LibvirtComputingResource","name":"cs-compute2.cloud.redacted.net","id":"2","version":"4.20.1.0","iqn":"iqn.1994-05.com.redhat:ce7749e17b2b","publicIpAddress":"10.25.0.3","publicNetmask":"255.255.255.0","publicMacAddress":"00:23:24:54:e2:1a","privateIpAddress":"10.25.0.3","privateMacAddress":"00:23:24:54:e2:1a","privateNetmask":"255.255.255.0","storageIpAddress":"10.25.0.3","storageNetmask":"255.255.255.0","storageMacAddress":"00:23:24:54:e2:1a","resourceName":"LibvirtComputingResource","gatewayIpAddress":"10.25.0.1","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d","host":"10.25.0.3","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(153.39 GB) 164705353728","availableBytes":"(46.80 GB) 50255364096"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"42055e9b-5ec7-3a35-9818-606ee1da20ea-LibvirtComputingResource","name":"cs-compute2.cloud.redacted.net","id":"2","version":"4.20.1.0","resourceName":"LibvirtComputingResource","msHostList":"172.16.20.40@static","connectionTransferred":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,931 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 4-3938116399158788100: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.storage.ListVolumeAnswer":{"secUrl":"nfs://nas.redacted.net/volume1/cs-secondary","templateInfos":{},"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,931 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:[]) (logid:) Seq 1-3938116399158788100: No more commands found 2025-06-29 03:36:46,931 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788100: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 110, { ListVolumeAnswer } } 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to CiscoVnmcElement 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp 2025-06-29 03:36:46,945 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer 2025-06-29 03:36:46,946 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) createHostVOForConnectedAgent: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:46,946 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl 2025-06-29 03:36:46,946 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovs 2025-06-29 03:36:46,946 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to VmwareServerDiscoverer 2025-06-29 03:36:46,946 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NsxElement 2025-06-29 03:36:46,946 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl 2025-06-29 03:36:46,946 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer 2025-06-29 03:36:46,948 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) getCommandHostDelegation: class com.cloud.agent.api.storage.ListTemplateCommand 2025-06-29 03:36:46,948 DEBUG [c.c.h.XenServerGuru] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:46,948 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.storage.ListTemplateCommand is 1800 seconds 2025-06-29 03:36:46,950 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 1-3938116399158788101: Routed from 90520738888109 2025-06-29 03:36:46,951 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788101: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.ListTemplateCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://nas.redacted.net/volume1/cs-secondary","_role":"Image","nfsVersion":"4"}},"nfsVersion":"4","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,966 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:[]) (logid:) Seq 1-4725683383995203587: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ModifyStoragePoolAnswer":{"poolInfo":{"host":"nas.redacted.net","localPath":"/mnt//10805e0a-2205-3158-993f-9350b08b9137","hostPath":"/volume1/cs-primary","poolType":"NetworkFilesystem","capacityBytes":"(55.8441 TB) 61401196134400","availableBytes":"(22.6087 TB) 24858509049856"},"templateInfo":{},"datastoreClusterChildren":[],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:46,967 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203587: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { ModifyStoragePoolAnswer } } 2025-06-29 03:36:47,010 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Connection established between storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} and host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,016 DEBUG [c.c.h.d.HostDaoImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Resource state update: [id = 2; name = cs-compute2.cloud.redacted.net; old state = Enabled; event = InternalCreated; new state = Enabled] 2025-06-29 03:36:47,016 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) [Resource state = Enabled, Agent event = , Host = AgentConnected] 2025-06-29 03:36:47,026 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:36:47,027 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:36:47,027 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:36:47,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) create ClusteredAgentAttache for Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: XcpServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,038 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Not XenServer so moving on. 2025-06-29 03:36:47,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: HypervServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,039 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Not Hyper-V hypervisor, so moving on. 2025-06-29 03:36:47,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: TungstenElement, for rebalance: false 2025-06-29 03:36:47,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,041 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: NsxElement, for rebalance: false 2025-06-29 03:36:47,042 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,042 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,042 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: ClusteredVirtualMachineManagerImpl, for rebalance: false 2025-06-29 03:36:47,042 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,042 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Received startup command from hypervisor host. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,045 INFO [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Reset VM power state sync for host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,053 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:36:47,055 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} connected, connecting host to shared pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} and sending storage pool information ... 2025-06-29 03:36:47,059 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Adding pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} to host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,065 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,065 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: SecurityGroupListener, for rebalance: false 2025-06-29 03:36:47,065 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,065 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Received a host startup notification 2025-06-29 03:36:47,067 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361857: Routed from 90520738888109 2025-06-29 03:36:47,068 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361857: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":"2378","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,068 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Scheduled network rules cleanup, interval=2378 2025-06-29 03:36:47,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,068 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Received a host startup notification 2025-06-29 03:36:47,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: SecondaryStorageListener, for rebalance: false 2025-06-29 03:36:47,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: DeploymentPlanningManagerImpl, for rebalance: false 2025-06-29 03:36:47,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,067 DEBUG [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending modify storage pool command to agent: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} for storage pool: StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} with timeout 300 seconds 2025-06-29 03:36:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: VmwareManagerImpl, for rebalance: false 2025-06-29 03:36:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: NetworkOrchestrator, for rebalance: false 2025-06-29 03:36:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,072 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 2-3926857400090361857: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,073 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Wait time setting on com.cloud.agent.api.ModifyStoragePoolCommand is 300 seconds 2025-06-29 03:36:47,075 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203588: Routed from 90520738888109 2025-06-29 03:36:47,076 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Host's hypervisorType is: KVM 2025-06-29 03:36:47,077 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203588: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"5","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80","host":"nas.redacted.net","path":"/volume2/cs-primary-ssd","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//80a947f2-ab02-3d8c-b1c0-7ba964ae0e80","details":{},"wait":"300","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,087 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending CheckNetworkCommand to check the Network is setup correctly on Agent 2025-06-29 03:36:47,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Wait time setting on com.cloud.agent.api.CheckNetworkCommand is 1800 seconds 2025-06-29 03:36:47,092 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361858: Routed from 90520738888109 2025-06-29 03:36:47,093 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361858: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.CheckNetworkCommand":{"networkInfoList":[{"physicalNetworkId":"200","privateNetworkName":"cloudbr0","publicNetworkName":"cloudbr0","guestNetworkName":"cloudbr0","storageNetworkName":"cloudbr0"}],"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,127 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:[]) (logid:) Seq 1-4725683383995203588: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ModifyStoragePoolAnswer":{"poolInfo":{"host":"nas.redacted.net","localPath":"/mnt//80a947f2-ab02-3d8c-b1c0-7ba964ae0e80","hostPath":"/volume2/cs-primary-ssd","poolType":"NetworkFilesystem","capacityBytes":"(1.2900 TB) 1418370088960","availableBytes":"(987.58 GB) 1060404330496"},"templateInfo":{},"datastoreClusterChildren":[],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,127 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203588: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { ModifyStoragePoolAnswer } } 2025-06-29 03:36:47,143 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Connection established between storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} and host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,153 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:36:47,154 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:36:47,155 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:36:47,170 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:36:47,174 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Removing pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} from host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,180 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 2-3926857400090361858: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.CheckNetworkAnswer":{"_reconnect":"false","result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,180 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:[]) (logid:) Seq 2-3926857400090361858: No more commands found 2025-06-29 03:36:47,181 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361858: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 110, { CheckNetworkAnswer } } 2025-06-29 03:36:47,181 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Network setup is correct on Agent 2025-06-29 03:36:47,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: StoragePoolMonitor, for rebalance: false 2025-06-29 03:36:47,182 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,187 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds 2025-06-29 03:36:47,189 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203589: Routed from 90520738888109 2025-06-29 03:36:47,191 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203589: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"12","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa","host":"10.25.0.2","path":"/var/lib/libvirt/images","port":"0","type":"Filesystem"},"_localPath":"/mnt//36cfe92b-5de4-3baf-9813-796fbddeb8af","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,205 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} connected, connecting host to shared pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} and sending storage pool information ... 2025-06-29 03:36:47,210 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Adding pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} to host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,221 DEBUG [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending modify storage pool command to agent: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} for storage pool: StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} with timeout 300 seconds 2025-06-29 03:36:47,225 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Wait time setting on com.cloud.agent.api.ModifyStoragePoolCommand is 300 seconds 2025-06-29 03:36:47,228 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361859: Routed from 90520738888109 2025-06-29 03:36:47,229 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361859: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"4","uuid":"10805e0a-2205-3158-993f-9350b08b9137","host":"nas.redacted.net","path":"/volume1/cs-primary","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//10805e0a-2205-3158-993f-9350b08b9137","details":{},"wait":"300","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,238 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 1-4725683383995203589: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,238 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203589: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { Answer } } 2025-06-29 03:36:47,251 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Connection removed between storage pool: StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} and host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: LocalStoragePoolListener, for rebalance: false 2025-06-29 03:36:47,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,287 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 2-3926857400090361859: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ModifyStoragePoolAnswer":{"poolInfo":{"host":"nas.redacted.net","localPath":"/mnt//10805e0a-2205-3158-993f-9350b08b9137","hostPath":"/volume1/cs-primary","poolType":"NetworkFilesystem","capacityBytes":"(55.8441 TB) 61401196134400","availableBytes":"(22.6087 TB) 24858509574144"},"templateInfo":{},"datastoreClusterChildren":[],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,288 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361859: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { ModifyStoragePoolAnswer } } 2025-06-29 03:36:47,298 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:36:47,300 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:36:47,300 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:36:47,323 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Connection established between storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} and host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,328 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:36:47,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: StorageCapacityListener, for rebalance: false 2025-06-29 03:36:47,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: ComputeCapacityListener, for rebalance: false 2025-06-29 03:36:47,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,340 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,341 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:36:47,342 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:36:47,343 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,345 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:36:47,352 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:36:47,353 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} connected, connecting host to shared pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} and sending storage pool information ... 2025-06-29 03:36:47,358 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Adding pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} to host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,371 DEBUG [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending modify storage pool command to agent: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} for storage pool: StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} with timeout 300 seconds 2025-06-29 03:36:47,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Wait time setting on com.cloud.agent.api.ModifyStoragePoolCommand is 300 seconds 2025-06-29 03:36:47,380 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361860: Routed from 90520738888109 2025-06-29 03:36:47,382 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361860: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"5","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80","host":"nas.redacted.net","path":"/volume2/cs-primary-ssd","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//80a947f2-ab02-3d8c-b1c0-7ba964ae0e80","details":{},"wait":"300","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,390 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:36:47,405 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:36:47,407 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:36:47,408 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:36:47,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: DirectNetworkStatsListener, for rebalance: false 2025-06-29 03:36:47,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: BehindOnPingListener, for rebalance: false 2025-06-29 03:36:47,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: SetHostParamsListener, for rebalance: false 2025-06-29 03:36:47,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,430 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203590: Routed from 90520738888109 2025-06-29 03:36:47,432 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203590: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SetHostParamsCommand":{"params":{"tungsten.plugin.enable":"false","migratewait":"3600","reconcile.commands.enabled":"false","router.aggregation.command.each.timeout":"600"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,432 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:47,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,439 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-3926857400090361860: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ModifyStoragePoolAnswer":{"poolInfo":{"host":"nas.redacted.net","localPath":"/mnt//80a947f2-ab02-3d8c-b1c0-7ba964ae0e80","hostPath":"/volume2/cs-primary-ssd","poolType":"NetworkFilesystem","capacityBytes":"(1.2900 TB) 1418370088960","availableBytes":"(987.58 GB) 1060402888704"},"templateInfo":{},"datastoreClusterChildren":[],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,440 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361860: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { ModifyStoragePoolAnswer } } 2025-06-29 03:36:47,440 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203591: Routed from 90520738888109 2025-06-29 03:36:47,441 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203591: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifySshKeysCommand":{"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,442 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,442 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: VirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:47,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,445 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: ConsoleProxyListener, for rebalance: false 2025-06-29 03:36:47,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:47,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,455 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203592: Routed from 90520738888109 2025-06-29 03:36:47,456 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203592: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifySshKeysCommand":{"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,457 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,457 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: VpcVirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:47,457 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,460 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[]) (logid:) Seq 1-4725683383995203590: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,462 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-10:[]) (logid:) Seq 2-4725683383995203590: No more commands found 2025-06-29 03:36:47,462 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,463 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 1-4725683383995203590: Response is not processed: Seq 1-4725683383995203590: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,468 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: DownloadListener, for rebalance: false 2025-06-29 03:36:47,469 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,472 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 1-4725683383995203592: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,474 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 1-4725683383995203591: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: UploadListener, for rebalance: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: KvmServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: CustomServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Sending Connect to listener: LxcServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) process connection to issue: null for host: 1, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,462 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Connection established between storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} and host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,478 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Wait time setting on com.cloud.agent.api.ReadyCommand is 60 seconds 2025-06-29 03:36:47,481 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203593: Routed from 90520738888109 2025-06-29 03:36:47,482 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203593: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"1","hostUuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b","hostName":"cs-compute1.cloud.redacted.net","enableHumanReadableSizes":"true","wait":"60","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,487 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:36:47,488 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:36:47,488 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:36:47,494 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:36:47,496 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Removing pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} from host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,506 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds 2025-06-29 03:36:47,510 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361861: Routed from 90520738888109 2025-06-29 03:36:47,511 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361861: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"13","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d","host":"10.25.0.3","path":"/var/lib/libvirt/images","port":"0","type":"Filesystem"},"_localPath":"/mnt//492af6d4-b5d1-3dc8-8a50-040101d28230","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,562 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:[]) (logid:) Seq 2-3926857400090361861: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,563 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361861: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { Answer } } 2025-06-29 03:36:47,576 INFO [o.a.c.s.d.p.DefaultHostListener] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Connection removed between storage pool: StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} and host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,578 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,578 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: LocalStoragePoolListener, for rebalance: false 2025-06-29 03:36:47,578 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,578 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,606 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:36:47,608 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:36:47,608 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:36:47,629 DEBUG [c.c.s.StorageManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:36:47,636 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: StorageCapacityListener, for rebalance: false 2025-06-29 03:36:47,636 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,636 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,636 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: ComputeCapacityListener, for rebalance: false 2025-06-29 03:36:47,636 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,639 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,642 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,649 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 4-3938116399158788101: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NumberFormatException: For input string: "" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) at java.base/java.lang.Long.parseLong(Long.java:721) at java.base/java.lang.Long.parseLong(Long.java:836) at com.cloud.storage.template.TemplateLocation.getTemplateInfo(TemplateLocation.java:185) at org.apache.cloudstack.storage.template.DownloadManagerImpl.gatherTemplateInfo(DownloadManagerImpl.java:1067) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:2243) at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:312) at com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:62) at com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:58) at com.cloud.agent.Agent.processRequest(Agent.java:779) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1194) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) ","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,649 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-14:[]) (logid:) Seq 2-3938116399158788101: No more commands found 2025-06-29 03:36:47,649 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788101: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 110, { Answer } } 2025-06-29 03:36:47,649 DEBUG [o.a.c.s.i.TemplateServiceImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) can not list template for secondary storage host ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"} 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: UploadListener, for rebalance: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: KvmServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: CustomServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Sending Connect to listener: LxcServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) process connection to issue: null for host: 4, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,654 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.ReadyCommand is 60 seconds 2025-06-29 03:36:47,655 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 2-3938116399158788102: Routed from 90520738888109 2025-06-29 03:36:47,656 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788102: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"4","hostUuid":"cc5764b8-77a5-4aa3-8141-f1334e493361","hostName":"s-2-VM","enableHumanReadableSizes":"true","wait":"60","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,657 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:36:47,660 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:36:47,661 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:36:47,662 DEBUG [c.c.c.CapacityManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:36:47,671 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: DirectNetworkStatsListener, for rebalance: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: BehindOnPingListener, for rebalance: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: SetHostParamsListener, for rebalance: false 2025-06-29 03:36:47,672 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,677 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361862: Routed from 90520738888109 2025-06-29 03:36:47,678 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361862: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SetHostParamsCommand":{"params":{"tungsten.plugin.enable":"false","migratewait":"3600","reconcile.commands.enabled":"false","router.aggregation.command.each.timeout":"600"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:47,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,683 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361863: Routed from 90520738888109 2025-06-29 03:36:47,684 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361863: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifySshKeysCommand":{"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,684 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,684 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: VirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:47,684 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,686 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: ConsoleProxyListener, for rebalance: false 2025-06-29 03:36:47,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: SshKeysDistriMonitor, for rebalance: false 2025-06-29 03:36:47,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,694 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361864: Routed from 90520738888109 2025-06-29 03:36:47,694 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361864: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifySshKeysCommand":{"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,695 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,695 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: VpcVirtualNetworkApplianceManagerImpl, for rebalance: false 2025-06-29 03:36:47,695 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,697 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,699 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: DownloadListener, for rebalance: false 2025-06-29 03:36:47,699 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,699 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: UploadListener, for rebalance: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:[]) (logid:) Seq 2-3926857400090361862: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: KvmServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-15:[]) (logid:) Seq 2-3926857400090361862: No more commands found 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: CustomServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 2-3926857400090361862: Response is not processed: Seq 2-3926857400090361862: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Sending Connect to listener: LxcServerDiscoverer, for rebalance: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,700 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) process connection to issue: null for host: 2, forRebalance: false, connection transferred: false 2025-06-29 03:36:47,701 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 2-3926857400090361864: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,701 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-3926857400090361863: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,703 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:[]) (logid:) Seq 4-3938116399158788102: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Wait time setting on com.cloud.agent.api.ReadyCommand is 60 seconds 2025-06-29 03:36:47,703 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-3:[]) (logid:) Seq 2-3938116399158788102: No more commands found 2025-06-29 03:36:47,703 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788102: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:47,703 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) [Resource state = Enabled, Agent event = , Host = Ready] 2025-06-29 03:36:47,706 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361865: Routed from 90520738888109 2025-06-29 03:36:47,707 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361865: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"2","hostUuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154","hostName":"cs-compute2.cloud.redacted.net","enableHumanReadableSizes":"true","wait":"60","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,719 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Wait time setting on com.cloud.agent.api.ReadyCommand is 1800 seconds 2025-06-29 03:36:47,720 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 2-3938116399158788103: Routed from 90520738888109 2025-06-29 03:36:47,722 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788103: Sending { Cmd , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"4","hostUuid":"cc5764b8-77a5-4aa3-8141-f1334e493361","hostName":"s-2-VM","enableHumanReadableSizes":"true","arch":"x86_64","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,837 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 4-3938116399158788103: Processing: { Ans: , MgmtId: 90520738888109, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:47,837 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:[ctx-5cd011fd]) (logid:b386aba8) Seq 4-3938116399158788103: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:47,837 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:[]) (logid:) Seq 2-3938116399158788103: No more commands found 2025-06-29 03:36:47,864 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb23677]) (logid:3f19bfa6) No inactive management server node found 2025-06-29 03:36:47,864 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb23677]) (logid:3f19bfa6) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:48,017 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a34dd189]) (logid:e2335f8d) HA health check task is running... 2025-06-29 03:36:48,032 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-4:[ctx-363300a4]) (logid:750824fc) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:36:48,048 DEBUG [o.a.c.s.NfsMountManagerImpl] (BackgroundTaskPollManager-5:[ctx-662d9839]) (logid:f68a95ec) Executing command [sudo mount -t nfs -o vers=4 nas.redacted.net:/volume1/cs-secondary /var/cloudstack/mnt/90520738888109.6dc1f50c ]. 2025-06-29 03:36:48,070 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5f894b98]) (logid:b09db3bc) ClusterDRS.poll is being called at 2025-06-29T01:37:00+0000 2025-06-29 03:36:48,084 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-206352aa]) (logid:df11ddcd) Resetting hosts suitable for reconnect 2025-06-29 03:36:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-206352aa]) (logid:df11ddcd) Completed resetting hosts suitable for reconnect 2025-06-29 03:36:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-206352aa]) (logid:df11ddcd) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:36:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-206352aa]) (logid:df11ddcd) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:36:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-206352aa]) (logid:df11ddcd) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:36:48,100 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-206352aa]) (logid:df11ddcd) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:36:48,104 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5f894b98]) (logid:b09db3bc) Removed 0 old drs migration plans 2025-06-29 03:36:48,151 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:[]) (logid:) Seq 1-4725683383995203593: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"detailsMap":{"host.uefi.enable":"true"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:48,151 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-4:[]) (logid:) Seq 2-4725683383995203593: No more commands found 2025-06-29 03:36:48,153 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203593: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:48,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Got HOST_UEFI_ENABLE [true] for host [Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}]: 2025-06-29 03:36:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c2b34476]) (logid:f3726029) VM scheduler.poll is being called at 2025-06-29T01:37:00+0000 2025-06-29 03:36:48,158 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) [Resource state = Enabled, Agent event = , Host = Ready] 2025-06-29 03:36:48,164 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c2b34476]) (logid:f3726029) Got 0 scheduled jobs to be executed at 2025-06-29T01:37:00+0000 2025-06-29 03:36:48,170 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c2b34476]) (logid:f3726029) Cleaned up 0 VM scheduled job entries 2025-06-29 03:36:48,174 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-f2a4e07e]) (logid:b9245636) VM Operation Thread Running 2025-06-29 03:36:48,188 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Wait time setting on com.cloud.agent.api.ReadyCommand is 1800 seconds 2025-06-29 03:36:48,191 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 2-4725683383995203594: Routed from 90520738888109 2025-06-29 03:36:48,192 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203594: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"1","hostUuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b","hostName":"cs-compute1.cloud.redacted.net","enableHumanReadableSizes":"true","arch":"x86_64","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:48,444 DEBUG [o.a.c.s.NfsMountManagerImpl] (BackgroundTaskPollManager-5:[ctx-662d9839]) (logid:f68a95ec) Successfully executed process [4064805] for command [sudo mount -t nfs -o vers=4 nas.redacted.net:/volume1/cs-secondary /var/cloudstack/mnt/90520738888109.6dc1f50c ]. 2025-06-29 03:36:48,445 DEBUG [o.a.c.s.NfsMountManagerImpl] (BackgroundTaskPollManager-5:[ctx-662d9839]) (logid:f68a95ec) Executing command [sudo chmod 1777 /var/cloudstack/mnt/90520738888109.6dc1f50c ]. 2025-06-29 03:36:48,522 DEBUG [o.a.c.s.NfsMountManagerImpl] (BackgroundTaskPollManager-5:[ctx-662d9839]) (logid:f68a95ec) Successfully executed process [4064829] for command [sudo chmod 1777 /var/cloudstack/mnt/90520738888109.6dc1f50c ]. 2025-06-29 03:36:48,761 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 1-4725683383995203594: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"detailsMap":{"host.uefi.enable":"true"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:48,761 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-6:[]) (logid:) Seq 2-4725683383995203594: No more commands found 2025-06-29 03:36:48,762 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-3:[ctx-26b3b711]) (logid:4c5c4abb) Seq 1-4725683383995203594: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:49,366 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-764c597a]) (logid:1ebbca97) No inactive management server node found 2025-06-29 03:36:49,366 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-764c597a]) (logid:1ebbca97) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:50,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6e6f5f2]) (logid:abda04c8) No inactive management server node found 2025-06-29 03:36:50,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6e6f5f2]) (logid:abda04c8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:36:51,914 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 2-3926857400090361865: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"detailsMap":{},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:51,914 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-7:[]) (logid:) Seq 2-3926857400090361865: No more commands found 2025-06-29 03:36:51,914 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361865: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:51,914 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Got HOST_UEFI_ENABLE [null] for host [Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}]: 2025-06-29 03:36:51,915 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) [Resource state = Enabled, Agent event = , Host = Ready] 2025-06-29 03:36:51,932 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Wait time setting on com.cloud.agent.api.ReadyCommand is 1800 seconds 2025-06-29 03:36:51,934 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361866: Routed from 90520738888109 2025-06-29 03:36:51,935 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361866: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":"1","hostId":"2","hostUuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154","hostName":"cs-compute2.cloud.redacted.net","enableHumanReadableSizes":"true","arch":"x86_64","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:52,020 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a6eb0d0e]) (logid:9cf20628) HA health check task is running... 2025-06-29 03:36:52,381 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bf3b90a]) (logid:02aef869) No inactive management server node found 2025-06-29 03:36:52,381 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bf3b90a]) (logid:02aef869) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:53,162 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-e692edfc]) (logid:500aa7aa) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:36:53,183 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-e692edfc]) (logid:500aa7aa) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:36:53,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e7c27ad]) (logid:4fcb4926) No inactive management server node found 2025-06-29 03:36:53,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e7c27ad]) (logid:4fcb4926) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:55,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8244: Seq 7-8244: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:55,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8244 2025-06-29 03:36:55,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3b75475]) (logid:b3725c57) No inactive management server node found 2025-06-29 03:36:55,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3b75475]) (logid:b3725c57) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:36:56,022 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ddfe0bf2]) (logid:82e79abe) HA health check task is running... 2025-06-29 03:36:56,114 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-3926857400090361866: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"detailsMap":{},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:36:56,114 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-9:[]) (logid:) Seq 2-3926857400090361866: No more commands found 2025-06-29 03:36:56,114 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-4:[ctx-d22dee82]) (logid:5fbc7c2a) Seq 2-3926857400090361866: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 110, { ReadyAnswer } } 2025-06-29 03:36:56,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fd31489]) (logid:7e396e51) No inactive management server node found 2025-06-29 03:36:56,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fd31489]) (logid:7e396e51) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:36:58,040 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:36:58,040 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-b1982403]) (logid:05dd0ec0) Starting to remove all volume_stats rows older than [Sat Jun 28 15:36:58 CEST 2025]. 2025-06-29 03:36:58,040 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-b22118be]) (logid:e255ff10) Starting to remove all vm_stats rows older than [Sat Jun 28 15:36:58 CEST 2025]. 2025-06-29 03:36:58,041 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:36:58,042 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-b1982403]) (logid:05dd0ec0) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:36:58 CEST 2025]. 2025-06-29 03:36:58,043 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:36:58,044 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) Seq 2-4725683383995203595: Routed from 90520738888109 2025-06-29 03:36:58,046 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-ae4a35d1]) (logid:41083902) AutoScaling Monitor is running... 2025-06-29 03:36:58,046 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) StorageCollector is running... 2025-06-29 03:36:58,057 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) Seq 1-4725683383995203595: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:36:58,058 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:36:58,058 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:58,058 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:36:58,061 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:36:58,066 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-3938116399158788104: Routed from 90520738888109 2025-06-29 03:36:58,068 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-b22118be]) (logid:e255ff10) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:36:58 CEST 2025]. 2025-06-29 03:36:58,068 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) Seq 2-3926857400090361867: Routed from 90520738888109 2025-06-29 03:36:58,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:36:58,071 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) Seq 2-4725683383995203596: Routed from 90520738888109 2025-06-29 03:36:58,084 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-2dbf2ba9]) (logid:c5ae09cb) Seq 2-3926857400090361867: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:36:58,125 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 4-3938116399158788104: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:36:58,135 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:36:58,135 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:58,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:36:58,136 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-4725683383995203597: Routed from 90520738888109 2025-06-29 03:36:58,142 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) Seq 1-4725683383995203596: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:36:58,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:36:58,158 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Seq 2-4725683383995203598: Routed from 90520738888109 2025-06-29 03:36:58,161 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 1-4725683383995203597: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:36:58,176 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:36:58,176 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:58,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:36:58,178 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-3926857400090361868: Routed from 90520738888109 2025-06-29 03:36:58,191 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:36:58,192 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) Seq 2-3926857400090361869: Routed from 90520738888109 2025-06-29 03:36:58,204 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Seq 1-4725683383995203598: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:36:58,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 03:36:58,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:36:58,256 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Seq 2-3926857400090361870: Routed from 90520738888109 2025-06-29 03:36:58,265 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Seq 2-3926857400090361870: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:36:58,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-28af41d3]) (logid:aae6bcea) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 03:36:58,287 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-3926857400090361868: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:36:58,299 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:36:58,299 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:58,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:36:58,301 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-4725683383995203599: Routed from 90520738888109 2025-06-29 03:36:58,322 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-093271b7]) (logid:b8a8917e) Seq 2-3926857400090361869: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:36:58,346 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 1-4725683383995203599: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:36:58,352 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:36:58,352 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:36:58,352 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:36:58,355 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-3926857400090361871: Routed from 90520738888109 2025-06-29 03:36:58,368 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b7f2c3]) (logid:87a7f544) No inactive management server node found 2025-06-29 03:36:58,368 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b7f2c3]) (logid:87a7f544) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:36:58,441 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-acee378c]) (logid:f157cd79) Seq 2-3926857400090361871: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:36:59,876 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2eca588]) (logid:92266b66) No inactive management server node found 2025-06-29 03:36:59,876 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2eca588]) (logid:92266b66) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 3ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:00,024 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-841a9f32]) (logid:ed377632) HA health check task is running... 2025-06-29 03:37:01,376 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89ee5c6b]) (logid:d28d33c0) No inactive management server node found 2025-06-29 03:37:01,376 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89ee5c6b]) (logid:d28d33c0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:02,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48b90d39]) (logid:b0b7eabf) No inactive management server node found 2025-06-29 03:37:02,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48b90d39]) (logid:b0b7eabf) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:04,026 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-313665a5]) (logid:44bbd2e7) HA health check task is running... 2025-06-29 03:37:04,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e90673b]) (logid:f3e61bd5) No inactive management server node found 2025-06-29 03:37:04,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e90673b]) (logid:f3e61bd5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:05,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8245: Seq 7-8245: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:05,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8245 2025-06-29 03:37:05,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f799eb92]) (logid:aafdb62c) No inactive management server node found 2025-06-29 03:37:05,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f799eb92]) (logid:aafdb62c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:07,378 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5935e9af]) (logid:ac2a8512) No inactive management server node found 2025-06-29 03:37:07,378 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5935e9af]) (logid:ac2a8512) Peer scan is finished. profiler: Done. Duration: 6ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:08,028 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3aedffb8]) (logid:2f75d18a) HA health check task is running... 2025-06-29 03:37:08,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59ffc9a1]) (logid:2028144b) No inactive management server node found 2025-06-29 03:37:08,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59ffc9a1]) (logid:2028144b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:10,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13e99eb5]) (logid:627c6290) No inactive management server node found 2025-06-29 03:37:10,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13e99eb5]) (logid:627c6290) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:11,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1867a1a]) (logid:f5d9afe0) No inactive management server node found 2025-06-29 03:37:11,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1867a1a]) (logid:f5d9afe0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:12,030 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5a0fffc1]) (logid:20f59cda) HA health check task is running... 2025-06-29 03:37:13,001 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ca4b2f3d]) (logid:f59a169a) Checking if any host reservation can be released ... 2025-06-29 03:37:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ca4b2f3d]) (logid:f59a169a) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:13,021 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ca4b2f3d]) (logid:f59a169a) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:13,021 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ca4b2f3d]) (logid:f59a169a) Done running HostReservationReleaseChecker ... 2025-06-29 03:37:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-87134b2c]) (logid:c7fb87b5) Found 0 routers to update status. 2025-06-29 03:37:13,103 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-87134b2c]) (logid:c7fb87b5) Found 0 VPC's to update Redundant State. 2025-06-29 03:37:13,105 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-87134b2c]) (logid:c7fb87b5) Found 0 networks to update RvR status. 2025-06-29 03:37:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Running Capacity Checker ... 2025-06-29 03:37:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) recalculating system capacity 2025-06-29 03:37:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Executing cpu/ram capacity update 2025-06-29 03:37:13,174 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:13,176 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:13,196 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:37:13,197 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:37:13,197 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:37:13,224 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:13,225 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:13,234 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:13,236 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:37:13,237 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:37:13,238 DEBUG [c.c.c.CapacityManagerImpl] (pool-12-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:37:13,250 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Done executing cpu/ram capacity update 2025-06-29 03:37:13,251 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Executing storage capacity update 2025-06-29 03:37:13,271 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:37:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:37:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:37:13,286 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:37:13,294 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:37:13,295 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:37:13,295 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:37:13,306 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:37:13,315 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:37:13,316 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:37:13,316 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:37:13,333 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:37:13,343 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:37:13,344 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:37:13,344 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:37:13,365 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58fa231d]) (logid:b79e114b) No inactive management server node found 2025-06-29 03:37:13,365 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58fa231d]) (logid:b79e114b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:13,369 DEBUG [c.c.s.StorageManagerImpl] (pool-13-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:37:13,370 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Done executing storage capacity update 2025-06-29 03:37:13,370 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Executing capacity updates for public ip and Vlans 2025-06-29 03:37:13,400 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Done capacity updates for public ip and Vlans 2025-06-29 03:37:13,400 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Executing capacity updates for private ip 2025-06-29 03:37:13,408 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Done executing capacity updates for private ip 2025-06-29 03:37:13,408 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Done recalculating system capacity 2025-06-29 03:37:13,469 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-7a3950b1]) (logid:3c3a18fe) Done running Capacity Checker ... 2025-06-29 03:37:14,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97286f7a]) (logid:cac18367) No inactive management server node found 2025-06-29 03:37:14,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97286f7a]) (logid:cac18367) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:15,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8246: Seq 7-8246: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:15,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8246 2025-06-29 03:37:16,032 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f7d9be8c]) (logid:43c2b72a) HA health check task is running... 2025-06-29 03:37:16,376 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f18813b1]) (logid:d5ff8432) No inactive management server node found 2025-06-29 03:37:16,376 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f18813b1]) (logid:d5ff8432) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:17,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-baa9f5ed]) (logid:ad5c0fe7) No inactive management server node found 2025-06-29 03:37:17,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-baa9f5ed]) (logid:ad5c0fe7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:19,378 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbbb856e]) (logid:17da46b8) No inactive management server node found 2025-06-29 03:37:19,378 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbbb856e]) (logid:17da46b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:20,033 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-08aae2a2]) (logid:732a2845) HA health check task is running... 2025-06-29 03:37:20,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fe3f4e]) (logid:525b530f) No inactive management server node found 2025-06-29 03:37:20,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fe3f4e]) (logid:525b530f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:22,380 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50cf5f5a]) (logid:d8c07ea3) No inactive management server node found 2025-06-29 03:37:22,380 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50cf5f5a]) (logid:d8c07ea3) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:23,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63ca3c92]) (logid:a51279e0) No inactive management server node found 2025-06-29 03:37:23,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63ca3c92]) (logid:a51279e0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:24,035 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5f8ba5b3]) (logid:fcefb9a2) HA health check task is running... 2025-06-29 03:37:25,210 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8247: Seq 7-8247: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:25,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8247 2025-06-29 03:37:25,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec8a4ebc]) (logid:668c996b) No inactive management server node found 2025-06-29 03:37:25,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec8a4ebc]) (logid:668c996b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6e363b7]) (logid:6398c3a1) No inactive management server node found 2025-06-29 03:37:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6e363b7]) (logid:6398c3a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:28,037 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e199b55d]) (logid:c6cf5b4d) HA health check task is running... 2025-06-29 03:37:28,106 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e74e0a7f]) (logid:0b88677f) Enabled non-edge zones available for scan: 1 2025-06-29 03:37:28,122 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e74e0a7f]) (logid:0b88677f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:37:28,126 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e74e0a7f]) (logid:0b88677f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:37:28,292 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-924c0f57]) (logid:f202d722) Enabled non-edge zones available for scan: 1 2025-06-29 03:37:28,303 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-924c0f57]) (logid:f202d722) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:37:28,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2a53fa3]) (logid:1984c023) No inactive management server node found 2025-06-29 03:37:28,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2a53fa3]) (logid:1984c023) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:29,876 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a319888]) (logid:8a6d5b27) No inactive management server node found 2025-06-29 03:37:29,876 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a319888]) (logid:8a6d5b27) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms 2025-06-29 03:37:30,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8248: Seq 7-8248: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:30,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8248 2025-06-29 03:37:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-375b366e]) (logid:c360e5da) No inactive management server node found 2025-06-29 03:37:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-375b366e]) (logid:c360e5da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:32,039 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-77859bc1]) (logid:f85b2714) HA health check task is running... 2025-06-29 03:37:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02667b6b]) (logid:2256a108) No inactive management server node found 2025-06-29 03:37:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02667b6b]) (logid:2256a108) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:34,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71d70d2c]) (logid:c629e2a7) No inactive management server node found 2025-06-29 03:37:34,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71d70d2c]) (logid:c629e2a7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:35,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3cedc47]) (logid:1015ad25) No inactive management server node found 2025-06-29 03:37:35,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3cedc47]) (logid:1015ad25) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:36,041 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e7fe044c]) (logid:b1c0a2b6) HA health check task is running... 2025-06-29 03:37:37,379 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a43c8e9]) (logid:275c4d56) No inactive management server node found 2025-06-29 03:37:37,379 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a43c8e9]) (logid:275c4d56) Peer scan is finished. profiler: Done. Duration: 8ms , profilerQueryActiveList: Done. Duration: 5ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:38,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f438180]) (logid:e6e38384) No inactive management server node found 2025-06-29 03:37:38,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f438180]) (logid:e6e38384) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:40,043 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cf859dea]) (logid:90ca9e99) HA health check task is running... 2025-06-29 03:37:40,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8249: Seq 7-8249: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:40,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8249 2025-06-29 03:37:40,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95b15a9c]) (logid:f35feb53) No inactive management server node found 2025-06-29 03:37:40,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95b15a9c]) (logid:f35feb53) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d11331fb]) (logid:e10d0a7e) No inactive management server node found 2025-06-29 03:37:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d11331fb]) (logid:e10d0a7e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:43,001 DEBUG [o.a.c.s.o.B.BucketUsageTask] (Bucket-Usage-1:[ctx-378f36e1]) (logid:bb2cc150) Completed updating bucket usage for all object stores 2025-06-29 03:37:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-97988a97]) (logid:9f4729c5) DbCollector is running... 2025-06-29 03:37:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-24748bf2]) (logid:85534fab) Found 0 routers to update status. 2025-06-29 03:37:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-24748bf2]) (logid:85534fab) Found 0 VPC's to update Redundant State. 2025-06-29 03:37:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-24748bf2]) (logid:85534fab) Found 0 networks to update RvR status. 2025-06-29 03:37:43,309 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-1:[]) (logid:) Starting work 2025-06-29 03:37:43,309 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[]) (logid:) Starting work 2025-06-29 03:37:43,310 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[]) (logid:) Starting work 2025-06-29 03:37:43,310 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-3:[]) (logid:) Starting work 2025-06-29 03:37:43,311 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-2:[]) (logid:) Starting work 2025-06-29 03:37:43,337 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Processing work HAWork[94-HA-303-Running-Investigating] 2025-06-29 03:37:43,343 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) RESTART with HAWORK 2025-06-29 03:37:43,351 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) HA on VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} 2025-06-29 03:37:43,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Wait time setting on com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds 2025-06-29 03:37:43,359 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Seq 2-4725683383995203600: Routed from 90520738888109 2025-06-29 03:37:43,360 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Seq 1-4725683383995203600: Sending { Cmd , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckVirtualMachineCommand":{"vmName":"i-5-303-VM","wait":"20","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:43,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dde07fd3]) (logid:0eca5684) No inactive management server node found 2025-06-29 03:37:43,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dde07fd3]) (logid:0eca5684) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:43,408 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 1-4725683383995203600: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.CheckVirtualMachineAnswer":{"vncPort":"0","state":"PowerOn","result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:43,408 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Seq 1-4725683383995203600: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { CheckVirtualMachineAnswer } } 2025-06-29 03:37:43,409 DEBUG [c.c.h.CheckOnAgentInvestigator] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Agent responded with state PowerOn 2025-06-29 03:37:43,410 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) SimpleInvestigator found VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} to be alive? true 2025-06-29 03:37:43,410 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) VM VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} is found to be alive by SimpleInvestigator 2025-06-29 03:37:43,411 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} is alive and host is up. No need to restart it. 2025-06-29 03:37:43,415 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-f3f862ec, work-94]) (logid:7e26f7b9) Completed work HAWork[94-HA-303-Running-Investigating]. Took 2/5 attempts. 2025-06-29 03:37:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) ManagementServerCollector is running... 2025-06-29 03:37:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) DB properties were already loaded 2025-06-29 03:37:44,042 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:37:44,044 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5d6594fd]) (logid:534eba68) HA health check task is running... 2025-06-29 03:37:44,062 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Execution of process [4064843] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:37:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Exit value of process [4064843] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:37:44,062 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Process [4064843] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:37:44,062 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) usage install: null 2025-06-29 03:37:44,062 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Sessions active: 0 2025-06-29 03:37:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:37:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064846] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:37:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:37:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064851] for command [/bin/bash -c uname -r ]. 2025-06-29 03:37:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:37:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064852] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:37:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) system memory from /proc: 4887515136 2025-06-29 03:37:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:37:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064857] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:37:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) free memory from /proc: 165126144 2025-06-29 03:37:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:37:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064862] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:37:44,110 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) used memory from /proc: 1069972 2025-06-29 03:37:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:37:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064863] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:37:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:37:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064864] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:37:44,131 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:37:44,138 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064865] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:37:44,138 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:37:44,141 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064872] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:37:44,141 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:37:44,146 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Successfully executed process [4064873] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:37:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail buffersdirect.capacity, 208451 2025-06-29 03:37:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail buffersdirect.count, 54 2025-06-29 03:37:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail buffersdirect.used, 208451 2025-06-29 03:37:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail buffersmapped.capacity, 0 2025-06-29 03:37:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail buffersmapped.count, 0 2025-06-29 03:37:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail buffersmapped.used, 0 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail jvmuptime, 123854 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:37:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memoryheap.committed, 517996544 2025-06-29 03:37:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memoryheap.init, 77594624 2025-06-29 03:37:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memoryheap.usage, 0.1548906560456219 2025-06-29 03:37:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorynon-heap.committed, 184745984 2025-06-29 03:37:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:37:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorynon-heap.max, -1 2025-06-29 03:37:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorynon-heap.usage, -1.8272168E8 2025-06-29 03:37:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorynon-heap.used, 182721680 2025-06-29 03:37:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30289473684210527 2025-06-29 03:37:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.08071701712876567 2025-06-29 03:37:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19082839004232346 2025-06-29 03:37:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012099802494049072 2025-06-29 03:37:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.Metaspace.usage, 0.9949711052946342 2025-06-29 03:37:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.PS-Eden-Space.usage, 0.16980049213093323 2025-06-29 03:37:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:37:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:37:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorytotal.committed, 702742528 2025-06-29 03:37:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorytotal.init, 85262336 2025-06-29 03:37:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorytotal.max, 1908932607 2025-06-29 03:37:44,155 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail memorytotal.used, 478397504 2025-06-29 03:37:44,156 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail threadsdeadlocks, [] 2025-06-29 03:37:44,157 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail threadsnew.count, 0 2025-06-29 03:37:44,158 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) not storing detail threadstimed_waiting.count, 91 2025-06-29 03:37:44,162 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-945c944b]) (logid:07d01fa0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":223131120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":123766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":477538808,"totalCommitted":702676992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295675824,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":165126144,"systemMemoryUsed":1069972,"systemMemoryVirtualSize":6218215424,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.49,0.27],"systemCyclesUsage":[66394342,53819986,4588667135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:37:44,163 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 3, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":223131120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":123766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":477538808,"totalCommitted":702676992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295675824,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":165126144,"systemMemoryUsed":1069972,"systemMemoryVirtualSize":6218215424,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.49,0.27],"systemCyclesUsage":[66394342,53819986,4588667135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:37:44,163 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:37:44,176 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-7:[ctx-1901fbdf]) (logid:a7202080) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":223131120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":123766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":477538808,"totalCommitted":702676992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295675824,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":165126144,"systemMemoryUsed":1069972,"systemMemoryVirtualSize":6218215424,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.49,0.27],"systemCyclesUsage":[66394342,53819986,4588667135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:37:44,179 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 14 ms 2025-06-29 03:37:44,179 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 15ms. agent: 0, pdu seq: 3, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":223131120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":123766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":477538808,"totalCommitted":702676992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295675824,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":17,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":165126144,"systemMemoryUsed":1069972,"systemMemoryVirtualSize":6218215424,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.49,0.27],"systemCyclesUsage":[66394342,53819986,4588667135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:37:44,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5cb8492]) (logid:f5a9767e) No inactive management server node found 2025-06-29 03:37:44,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5cb8492]) (logid:f5a9767e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:45,764 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:37:45,926 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:37:46,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-357b9d42]) (logid:fc05ccd8) No inactive management server node found 2025-06-29 03:37:46,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-357b9d42]) (logid:fc05ccd8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:46,656 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:37:46,685 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":50,"instanceName":"i-5-50-VM","state":"Running","type":"User","uuid":"1ef8b8a0-f773-4867-b491-342785c57e98"}, power state: PowerOn 2025-06-29 03:37:46,703 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-14:[]) (logid:) VM instance {"id":50,"instanceName":"i-5-50-VM","state":"Running","type":"User","uuid":"1ef8b8a0-f773-4867-b491-342785c57e98"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:46,705 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":74,"instanceName":"i-5-74-VM","state":"Running","type":"User","uuid":"01b29297-2d6d-4fe9-9006-7c66f30d5043"}, power state: PowerOn 2025-06-29 03:37:46,721 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-14:[]) (logid:) VM instance {"id":74,"instanceName":"i-5-74-VM","state":"Running","type":"User","uuid":"01b29297-2d6d-4fe9-9006-7c66f30d5043"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:46,723 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"}, power state: PowerOn 2025-06-29 03:37:46,736 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-14:[]) (logid:) VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:46,739 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:37:47,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,087 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:37:47,103 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":2,"instanceName":"s-2-VM","state":"Running","type":"SecondaryStorageVm","uuid":"9e5a7cce-ef20-426e-b6fe-c076abf53264"}, power state: PowerOn 2025-06-29 03:37:47,115 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-15:[]) (logid:) VM instance {"id":2,"instanceName":"s-2-VM","state":"Running","type":"SecondaryStorageVm","uuid":"9e5a7cce-ef20-426e-b6fe-c076abf53264"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,116 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":306,"instanceName":"v-306-VM","state":"Running","type":"ConsoleProxy","uuid":"d1ec8241-db64-4a9e-8f6f-866239a02a33"}, power state: PowerOn 2025-06-29 03:37:47,128 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-15:[]) (logid:) VM instance {"id":306,"instanceName":"v-306-VM","state":"Running","type":"ConsoleProxy","uuid":"d1ec8241-db64-4a9e-8f6f-866239a02a33"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,130 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":110,"instanceName":"i-5-110-VM","state":"Running","type":"User","uuid":"fd1af66f-e131-42ef-9f31-42f4a54405d7"}, power state: PowerOn 2025-06-29 03:37:47,142 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-15:[]) (logid:) VM instance {"id":110,"instanceName":"i-5-110-VM","state":"Running","type":"User","uuid":"fd1af66f-e131-42ef-9f31-42f4a54405d7"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,144 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"}, power state: PowerOn 2025-06-29 03:37:47,157 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-15:[]) (logid:) VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,160 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":127,"instanceName":"i-5-127-VM","state":"Running","type":"User","uuid":"3fa040d9-feea-484c-aba4-3a7770ff6209"}, power state: PowerOn 2025-06-29 03:37:47,173 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-15:[]) (logid:) VM instance {"id":127,"instanceName":"i-5-127-VM","state":"Running","type":"User","uuid":"3fa040d9-feea-484c-aba4-3a7770ff6209"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,176 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:37:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59252ddb]) (logid:f29e5ea7) No inactive management server node found 2025-06-29 03:37:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59252ddb]) (logid:f29e5ea7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:48,046 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-76036826]) (logid:de49b962) HA health check task is running... 2025-06-29 03:37:48,066 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-75f0630a]) (logid:bc53b1f3) ClusterDRS.poll is being called at 2025-06-29T01:38:00+0000 2025-06-29 03:37:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-75f0630a]) (logid:bc53b1f3) Removed 0 old drs migration plans 2025-06-29 03:37:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cfbd3ca7]) (logid:7fe6ca81) VM scheduler.poll is being called at 2025-06-29T01:38:00+0000 2025-06-29 03:37:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cfbd3ca7]) (logid:7fe6ca81) Got 0 scheduled jobs to be executed at 2025-06-29T01:38:00+0000 2025-06-29 03:37:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-cfbd3ca7]) (logid:7fe6ca81) Cleaned up 0 VM scheduled job entries 2025-06-29 03:37:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-42549c40]) (logid:5ea52638) VM Operation Thread Running 2025-06-29 03:37:49,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f3f2ef5]) (logid:8cf604df) No inactive management server node found 2025-06-29 03:37:49,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f3f2ef5]) (logid:8cf604df) Peer scan is finished. profiler: Done. Duration: 6ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:50,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8251: Seq 7-8251: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:37:50,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8251 2025-06-29 03:37:50,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8668fa57]) (logid:d734077c) No inactive management server node found 2025-06-29 03:37:50,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8668fa57]) (logid:d734077c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:52,048 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d4c88ef2]) (logid:3d8c80fe) HA health check task is running... 2025-06-29 03:37:52,376 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93aa7d22]) (logid:07186691) No inactive management server node found 2025-06-29 03:37:52,377 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93aa7d22]) (logid:07186691) Peer scan is finished. profiler: Done. Duration: 6ms , profilerQueryActiveList: Done. Duration: 5ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10360c3b]) (logid:054ba743) No inactive management server node found 2025-06-29 03:37:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10360c3b]) (logid:054ba743) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:55,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6167a2bf]) (logid:26b38efa) No inactive management server node found 2025-06-29 03:37:55,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6167a2bf]) (logid:26b38efa) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:37:56,050 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-27e54274]) (logid:6325ac90) HA health check task is running... 2025-06-29 03:37:56,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30470f82]) (logid:8b4fb399) No inactive management server node found 2025-06-29 03:37:56,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30470f82]) (logid:8b4fb399) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:58,045 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-7d9709ff]) (logid:cd2bc5e6) Starting to remove all volume_stats rows older than [Sat Jun 28 15:37:58 CEST 2025]. 2025-06-29 03:37:58,046 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-7d9709ff]) (logid:cd2bc5e6) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:37:58 CEST 2025]. 2025-06-29 03:37:58,054 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-76606f9b]) (logid:80ac443e) AutoScaling Monitor is running... 2025-06-29 03:37:58,069 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-f39ea066]) (logid:41c0934c) Starting to remove all vm_stats rows older than [Sat Jun 28 15:37:58 CEST 2025]. 2025-06-29 03:37:58,090 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:37:58,091 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-217f104b]) (logid:b858f7fb) Enabled non-edge zones available for scan: 1 2025-06-29 03:37:58,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:37:58,093 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) Seq 2-4725683383995203601: Routed from 90520738888109 2025-06-29 03:37:58,095 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-f39ea066]) (logid:41c0934c) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:37:58 CEST 2025]. 2025-06-29 03:37:58,097 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) Seq 1-4725683383995203601: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:37:58,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:37:58,099 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) Seq 2-3926857400090361872: Routed from 90520738888109 2025-06-29 03:37:58,099 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-217f104b]) (logid:b858f7fb) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:37:58,101 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-217f104b]) (logid:b858f7fb) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:37:58,104 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-12bb5197]) (logid:cfa7797a) Seq 2-3926857400090361872: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:37:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b174fc4b]) (logid:32b76435) Enabled non-edge zones available for scan: 1 2025-06-29 03:37:58,284 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b174fc4b]) (logid:32b76435) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:37:58,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9cd7226]) (logid:f206808e) No inactive management server node found 2025-06-29 03:37:58,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9cd7226]) (logid:f206808e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:37:58,400 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:37:58,405 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:37:58,407 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) Seq 2-4725683383995203602: Routed from 90520738888109 2025-06-29 03:37:58,442 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) StorageCollector is running... 2025-06-29 03:37:58,452 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:37:58,452 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:37:58,452 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:37:58,453 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-3938116399158788105: Routed from 90520738888109 2025-06-29 03:37:58,464 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) Seq 1-4725683383995203602: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:37:58,504 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:37:58,506 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) Seq 2-3926857400090361873: Routed from 90520738888109 2025-06-29 03:37:58,510 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 4-3938116399158788105: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:37:58,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:37:58,518 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:37:58,518 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:37:58,521 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-3926857400090361874: Routed from 90520738888109 2025-06-29 03:37:58,560 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-3926857400090361874: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:37:58,573 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:37:58,573 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:37:58,573 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:37:58,574 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-4725683383995203603: Routed from 90520738888109 2025-06-29 03:37:58,614 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-3fc1c794]) (logid:017d85b2) Seq 2-3926857400090361873: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:37:58,624 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 1-4725683383995203603: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:37:58,639 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:37:58,639 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:37:58,639 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:37:58,641 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-4725683383995203604: Routed from 90520738888109 2025-06-29 03:37:58,726 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 1-4725683383995203604: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:37:58,729 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:37:58,729 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:37:58,729 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:37:58,730 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-3926857400090361875: Routed from 90520738888109 2025-06-29 03:37:58,775 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e51f2b63]) (logid:b9fdb9c3) Seq 2-3926857400090361875: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:37:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4fdbca3]) (logid:8c252d14) No inactive management server node found 2025-06-29 03:37:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4fdbca3]) (logid:8c252d14) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:00,051 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6854f636]) (logid:70a638f0) HA health check task is running... 2025-06-29 03:38:00,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8252: Seq 7-8252: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:00,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8252 2025-06-29 03:38:01,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a23fa51]) (logid:25753654) No inactive management server node found 2025-06-29 03:38:01,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a23fa51]) (logid:25753654) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d8d9a5d]) (logid:c569c91b) No inactive management server node found 2025-06-29 03:38:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d8d9a5d]) (logid:c569c91b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:04,053 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e9754b4d]) (logid:0b9272b7) HA health check task is running... 2025-06-29 03:38:04,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35391482]) (logid:3f00761f) No inactive management server node found 2025-06-29 03:38:04,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35391482]) (logid:3f00761f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:38:05,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e6586f0]) (logid:43e76381) No inactive management server node found 2025-06-29 03:38:05,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e6586f0]) (logid:43e76381) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:07,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0b64f65]) (logid:40979517) No inactive management server node found 2025-06-29 03:38:07,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0b64f65]) (logid:40979517) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:08,055 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c4387a1f]) (logid:2e32da09) HA health check task is running... 2025-06-29 03:38:08,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72c90a83]) (logid:2e1ddfc5) No inactive management server node found 2025-06-29 03:38:08,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72c90a83]) (logid:2e1ddfc5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:10,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8253: Seq 7-8253: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:10,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8253 2025-06-29 03:38:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcec8f75]) (logid:f5779a80) No inactive management server node found 2025-06-29 03:38:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcec8f75]) (logid:f5779a80) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9adc62b5]) (logid:65ad6acf) No inactive management server node found 2025-06-29 03:38:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9adc62b5]) (logid:65ad6acf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:12,057 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3ee68b1b]) (logid:1906ac4d) HA health check task is running... 2025-06-29 03:38:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bc2ee46d]) (logid:efbc96ad) Found 0 routers to update status. 2025-06-29 03:38:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bc2ee46d]) (logid:efbc96ad) Found 0 VPC's to update Redundant State. 2025-06-29 03:38:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bc2ee46d]) (logid:efbc96ad) Found 0 networks to update RvR status. 2025-06-29 03:38:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a96c0e87]) (logid:398da287) No inactive management server node found 2025-06-29 03:38:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a96c0e87]) (logid:398da287) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:14,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f21b5ac0]) (logid:cfc6d1d7) No inactive management server node found 2025-06-29 03:38:14,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f21b5ac0]) (logid:cfc6d1d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:16,059 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7a6580e9]) (logid:70373b61) HA health check task is running... 2025-06-29 03:38:16,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51efac5c]) (logid:97c8f81c) No inactive management server node found 2025-06-29 03:38:16,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51efac5c]) (logid:97c8f81c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:17,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32ba9342]) (logid:4f4d55a4) No inactive management server node found 2025-06-29 03:38:17,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32ba9342]) (logid:4f4d55a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:18,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e47096b7]) (logid:226bb2ab) Resetting hosts suitable for reconnect 2025-06-29 03:38:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e47096b7]) (logid:226bb2ab) Completed resetting hosts suitable for reconnect 2025-06-29 03:38:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e47096b7]) (logid:226bb2ab) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:38:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e47096b7]) (logid:226bb2ab) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:38:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e47096b7]) (logid:226bb2ab) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:38:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e47096b7]) (logid:226bb2ab) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:38:19,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db2bd977]) (logid:e725a487) No inactive management server node found 2025-06-29 03:38:19,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db2bd977]) (logid:e725a487) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:20,060 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-80cd9a45]) (logid:1c6e8db5) HA health check task is running... 2025-06-29 03:38:20,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8254: Seq 7-8254: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:20,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8254 2025-06-29 03:38:20,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af57da23]) (logid:ab899ae1) No inactive management server node found 2025-06-29 03:38:20,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af57da23]) (logid:ab899ae1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-748e639a]) (logid:d245af74) No inactive management server node found 2025-06-29 03:38:22,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-748e639a]) (logid:d245af74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:23,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f97308b]) (logid:5d7d9151) No inactive management server node found 2025-06-29 03:38:23,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f97308b]) (logid:5d7d9151) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:24,062 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-540cab68]) (logid:3e6d88a0) HA health check task is running... 2025-06-29 03:38:25,377 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3abe7451]) (logid:a0035d61) No inactive management server node found 2025-06-29 03:38:25,377 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3abe7451]) (logid:a0035d61) Peer scan is finished. profiler: Done. Duration: 6ms , profilerQueryActiveList: Done. Duration: 5ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:26,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-580cc27c]) (logid:3aee2978) No inactive management server node found 2025-06-29 03:38:26,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-580cc27c]) (logid:3aee2978) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:28,064 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5ee3003d]) (logid:5d6be31a) HA health check task is running... 2025-06-29 03:38:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b7c6cda8]) (logid:dbbe7fd1) Enabled non-edge zones available for scan: 1 2025-06-29 03:38:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b7c6cda8]) (logid:dbbe7fd1) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:38:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b7c6cda8]) (logid:dbbe7fd1) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:38:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-97170952]) (logid:61de5b85) Enabled non-edge zones available for scan: 1 2025-06-29 03:38:28,289 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-97170952]) (logid:61de5b85) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:38:28,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ab1abda]) (logid:dce03265) No inactive management server node found 2025-06-29 03:38:28,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ab1abda]) (logid:dce03265) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:29,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2206055c]) (logid:76417a17) No inactive management server node found 2025-06-29 03:38:29,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2206055c]) (logid:76417a17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:30,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8255: Seq 7-8255: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:30,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8255 2025-06-29 03:38:31,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d5a35eb]) (logid:1100d562) No inactive management server node found 2025-06-29 03:38:31,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d5a35eb]) (logid:1100d562) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:32,065 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-baa325fd]) (logid:a6811835) HA health check task is running... 2025-06-29 03:38:32,875 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d80fced]) (logid:934f9937) No inactive management server node found 2025-06-29 03:38:32,875 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d80fced]) (logid:934f9937) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:34,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9474a72e]) (logid:523b2764) No inactive management server node found 2025-06-29 03:38:34,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9474a72e]) (logid:523b2764) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:35,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8256: Seq 7-8256: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:35,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8256 2025-06-29 03:38:35,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71ed895f]) (logid:04a615ac) No inactive management server node found 2025-06-29 03:38:35,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71ed895f]) (logid:04a615ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:36,067 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dd63b1b2]) (logid:436a7276) HA health check task is running... 2025-06-29 03:38:37,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab43c949]) (logid:d029fb52) No inactive management server node found 2025-06-29 03:38:37,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab43c949]) (logid:d029fb52) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:38,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a428123a]) (logid:1f556f29) No inactive management server node found 2025-06-29 03:38:38,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a428123a]) (logid:1f556f29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:40,069 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-be1c1c87]) (logid:5ffcd71a) HA health check task is running... 2025-06-29 03:38:40,376 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-071a5a17]) (logid:eb791aed) No inactive management server node found 2025-06-29 03:38:40,376 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-071a5a17]) (logid:eb791aed) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 3ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1f9bd5b]) (logid:d8205e52) No inactive management server node found 2025-06-29 03:38:41,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1f9bd5b]) (logid:d8205e52) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-cd6675b4]) (logid:5f057f8a) DbCollector is running... 2025-06-29 03:38:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2829106b]) (logid:754de60d) Found 0 routers to update status. 2025-06-29 03:38:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2829106b]) (logid:754de60d) Found 0 VPC's to update Redundant State. 2025-06-29 03:38:43,103 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2829106b]) (logid:754de60d) Found 0 networks to update RvR status. 2025-06-29 03:38:43,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a6a28dd]) (logid:276014e7) No inactive management server node found 2025-06-29 03:38:43,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a6a28dd]) (logid:276014e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) ManagementServerCollector is running... 2025-06-29 03:38:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) DB properties were already loaded 2025-06-29 03:38:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:38:44,057 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Execution of process [4064886] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:38:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Exit value of process [4064886] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:38:44,057 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Process [4064886] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:38:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) usage install: null 2025-06-29 03:38:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Sessions active: 0 2025-06-29 03:38:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:38:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064889] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:38:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:38:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064894] for command [/bin/bash -c uname -r ]. 2025-06-29 03:38:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:38:44,070 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f9d716fc]) (logid:003c19f3) HA health check task is running... 2025-06-29 03:38:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064895] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:38:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) system memory from /proc: 4887515136 2025-06-29 03:38:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:38:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064900] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:38:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) free memory from /proc: 159555584 2025-06-29 03:38:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:38:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064905] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:38:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) used memory from /proc: 1070560 2025-06-29 03:38:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:38:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064906] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:38:44,102 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:38:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064907] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:38:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:38:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064908] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:38:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:38:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064915] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:38:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:38:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Successfully executed process [4064916] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:38:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail buffersdirect.capacity, 149267 2025-06-29 03:38:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail buffersdirect.count, 53 2025-06-29 03:38:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail buffersdirect.used, 149267 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail buffersmapped.capacity, 0 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail buffersmapped.count, 0 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail buffersmapped.used, 0 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:38:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail jvmuptime, 183837 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memoryheap.committed, 517996544 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memoryheap.init, 77594624 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memoryheap.usage, 0.17316723210377472 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorynon-heap.committed, 185270272 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorynon-heap.max, -1 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorynon-heap.usage, -1.81122624E8 2025-06-29 03:38:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorynon-heap.used, 181122624 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30289473684210527 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.0795975239936017 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.17724822541407004 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.Compressed-Class-Space.usage, 0.01211555302143097 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.Metaspace.usage, 0.9949458047971094 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.PS-Eden-Space.usage, 0.22333630341631888 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:38:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:38:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorytotal.committed, 703266816 2025-06-29 03:38:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorytotal.init, 85262336 2025-06-29 03:38:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorytotal.max, 1908932607 2025-06-29 03:38:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail memorytotal.used, 511689856 2025-06-29 03:38:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail threadsdeadlocks, [] 2025-06-29 03:38:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail threadsnew.count, 0 2025-06-29 03:38:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) not storing detail threadstimed_waiting.count, 82 2025-06-29 03:38:44,141 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-8ad266c8]) (logid:ac55ad9c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":188255800,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":510847128,"totalCommitted":703266816,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":330564576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":192,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":159555584,"systemMemoryUsed":1070560,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.43,0.26],"systemCyclesUsage":[66394893,53820327,4588689985],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:38:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 5, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":188255800,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":510847128,"totalCommitted":703266816,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":330564576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":192,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":159555584,"systemMemoryUsed":1070560,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.43,0.26],"systemCyclesUsage":[66394893,53820327,4588689985],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:38:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:38:44,153 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-7:[ctx-f7df8223]) (logid:88233f07) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":188255800,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":510847128,"totalCommitted":703266816,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":330564576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":192,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":159555584,"systemMemoryUsed":1070560,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.43,0.26],"systemCyclesUsage":[66394893,53820327,4588689985],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:38:44,154 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 10 ms 2025-06-29 03:38:44,154 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 12ms. agent: 0, pdu seq: 5, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":188255800,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":510847128,"totalCommitted":703266816,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":330564576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":192,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":159555584,"systemMemoryUsed":1070560,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.43,0.26],"systemCyclesUsage":[66394893,53820327,4588689985],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:38:44,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a88453b7]) (logid:84730ef8) No inactive management server node found 2025-06-29 03:38:44,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a88453b7]) (logid:84730ef8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:45,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8257: Seq 7-8257: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:45,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8257 2025-06-29 03:38:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:38:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:38:46,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d2f301e]) (logid:2cc8f48c) No inactive management server node found 2025-06-29 03:38:46,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d2f301e]) (logid:2cc8f48c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:38:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:38:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:38:46,662 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":50,"instanceName":"i-5-50-VM","state":"Running","type":"User","uuid":"1ef8b8a0-f773-4867-b491-342785c57e98"}, power state: PowerOn 2025-06-29 03:38:46,675 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-5:[]) (logid:) VM instance {"id":50,"instanceName":"i-5-50-VM","state":"Running","type":"User","uuid":"1ef8b8a0-f773-4867-b491-342785c57e98"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:38:46,677 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":74,"instanceName":"i-5-74-VM","state":"Running","type":"User","uuid":"01b29297-2d6d-4fe9-9006-7c66f30d5043"}, power state: PowerOn 2025-06-29 03:38:46,689 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-5:[]) (logid:) VM instance {"id":74,"instanceName":"i-5-74-VM","state":"Running","type":"User","uuid":"01b29297-2d6d-4fe9-9006-7c66f30d5043"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:38:46,691 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"}, power state: PowerOn 2025-06-29 03:38:46,703 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-5:[]) (logid:) VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:38:46,704 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:38:47,094 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:38:47,111 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":2,"instanceName":"s-2-VM","state":"Running","type":"SecondaryStorageVm","uuid":"9e5a7cce-ef20-426e-b6fe-c076abf53264"}, power state: PowerOn 2025-06-29 03:38:47,123 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-4:[]) (logid:) VM instance {"id":2,"instanceName":"s-2-VM","state":"Running","type":"SecondaryStorageVm","uuid":"9e5a7cce-ef20-426e-b6fe-c076abf53264"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,124 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":306,"instanceName":"v-306-VM","state":"Running","type":"ConsoleProxy","uuid":"d1ec8241-db64-4a9e-8f6f-866239a02a33"}, power state: PowerOn 2025-06-29 03:38:47,134 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-4:[]) (logid:) VM instance {"id":306,"instanceName":"v-306-VM","state":"Running","type":"ConsoleProxy","uuid":"d1ec8241-db64-4a9e-8f6f-866239a02a33"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,136 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":110,"instanceName":"i-5-110-VM","state":"Running","type":"User","uuid":"fd1af66f-e131-42ef-9f31-42f4a54405d7"}, power state: PowerOn 2025-06-29 03:38:47,148 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-4:[]) (logid:) VM instance {"id":110,"instanceName":"i-5-110-VM","state":"Running","type":"User","uuid":"fd1af66f-e131-42ef-9f31-42f4a54405d7"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,149 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"}, power state: PowerOn 2025-06-29 03:38:47,162 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-4:[]) (logid:) VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,164 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":127,"instanceName":"i-5-127-VM","state":"Running","type":"User","uuid":"3fa040d9-feea-484c-aba4-3a7770ff6209"}, power state: PowerOn 2025-06-29 03:38:47,181 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-4:[]) (logid:) VM instance {"id":127,"instanceName":"i-5-127-VM","state":"Running","type":"User","uuid":"3fa040d9-feea-484c-aba4-3a7770ff6209"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,183 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:38:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33b8963f]) (logid:087d7f9f) No inactive management server node found 2025-06-29 03:38:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33b8963f]) (logid:087d7f9f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:48,066 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-10bd4bd8]) (logid:ecba8131) ClusterDRS.poll is being called at 2025-06-29T01:39:00+0000 2025-06-29 03:38:48,072 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-131d721d]) (logid:919c6147) HA health check task is running... 2025-06-29 03:38:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-10bd4bd8]) (logid:ecba8131) Removed 0 old drs migration plans 2025-06-29 03:38:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6cf6c12c]) (logid:70b7384c) VM scheduler.poll is being called at 2025-06-29T01:39:00+0000 2025-06-29 03:38:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6cf6c12c]) (logid:70b7384c) Got 0 scheduled jobs to be executed at 2025-06-29T01:39:00+0000 2025-06-29 03:38:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6cf6c12c]) (logid:70b7384c) Cleaned up 0 VM scheduled job entries 2025-06-29 03:38:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-66aca21b]) (logid:0fddb534) VM Operation Thread Running 2025-06-29 03:38:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7837be0b]) (logid:b9776af8) No inactive management server node found 2025-06-29 03:38:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7837be0b]) (logid:b9776af8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa00b699]) (logid:ba48c59e) No inactive management server node found 2025-06-29 03:38:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa00b699]) (logid:ba48c59e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:52,074 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8e5bd1ff]) (logid:624881ae) HA health check task is running... 2025-06-29 03:38:52,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0007d97e]) (logid:353585ff) No inactive management server node found 2025-06-29 03:38:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0007d97e]) (logid:353585ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0f75223]) (logid:9b4d300b) No inactive management server node found 2025-06-29 03:38:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0f75223]) (logid:9b4d300b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:55,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8259: Seq 7-8259: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:38:55,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8259 2025-06-29 03:38:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c50143e7]) (logid:5dee19d0) No inactive management server node found 2025-06-29 03:38:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c50143e7]) (logid:5dee19d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:56,076 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fb339c44]) (logid:4f7348b4) HA health check task is running... 2025-06-29 03:38:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c07a694]) (logid:ad20bcda) No inactive management server node found 2025-06-29 03:38:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c07a694]) (logid:ad20bcda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:58,049 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-297f7572]) (logid:1ee77194) Starting to remove all volume_stats rows older than [Sat Jun 28 15:38:58 CEST 2025]. 2025-06-29 03:38:58,049 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-297f7572]) (logid:1ee77194) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:38:58 CEST 2025]. 2025-06-29 03:38:58,057 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-f1801491]) (logid:8e17f684) AutoScaling Monitor is running... 2025-06-29 03:38:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-efa919f5]) (logid:7c2d65cc) Enabled non-edge zones available for scan: 1 2025-06-29 03:38:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-efa919f5]) (logid:7c2d65cc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:38:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-efa919f5]) (logid:7c2d65cc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:38:58,096 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-3137543b]) (logid:d5d66f04) Starting to remove all vm_stats rows older than [Sat Jun 28 15:38:58 CEST 2025]. 2025-06-29 03:38:58,108 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:38:58,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:38:58,111 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) Seq 2-4725683383995203605: Routed from 90520738888109 2025-06-29 03:38:58,114 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) Seq 1-4725683383995203605: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:38:58,117 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:38:58,118 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) Seq 2-3926857400090361876: Routed from 90520738888109 2025-06-29 03:38:58,122 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8b59e73a]) (logid:3ab5d622) Seq 2-3926857400090361876: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:38:58,136 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-3137543b]) (logid:d5d66f04) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:38:58 CEST 2025]. 2025-06-29 03:38:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-599246d3]) (logid:67f6808e) Enabled non-edge zones available for scan: 1 2025-06-29 03:38:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-599246d3]) (logid:67f6808e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:38:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c8a9da2]) (logid:c5d59876) No inactive management server node found 2025-06-29 03:38:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c8a9da2]) (logid:c5d59876) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:38:58,678 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:38:58,683 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:38:58,684 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) Seq 2-4725683383995203606: Routed from 90520738888109 2025-06-29 03:38:58,741 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) Seq 1-4725683383995203606: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:38:58,776 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) StorageCollector is running... 2025-06-29 03:38:58,789 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:38:58,789 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:38:58,789 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:38:58,791 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-3938116399158788106: Routed from 90520738888109 2025-06-29 03:38:58,791 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:38:58,792 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) Seq 2-3926857400090361877: Routed from 90520738888109 2025-06-29 03:38:58,844 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 4-3938116399158788106: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:38:58,849 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:38:58,849 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:38:58,849 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:38:58,850 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-4725683383995203607: Routed from 90520738888109 2025-06-29 03:38:58,887 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-1d0073b9]) (logid:8e062693) Seq 2-3926857400090361877: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:38:58,901 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 1-4725683383995203607: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:38:58,916 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:38:58,916 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:38:58,916 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:38:58,917 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-3926857400090361878: Routed from 90520738888109 2025-06-29 03:38:58,972 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-3926857400090361878: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:38:58,983 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:38:58,983 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:38:58,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:38:58,985 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-4725683383995203608: Routed from 90520738888109 2025-06-29 03:38:59,029 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 1-4725683383995203608: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:38:59,031 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:38:59,031 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:38:59,032 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:38:59,033 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-3926857400090361879: Routed from 90520738888109 2025-06-29 03:38:59,079 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2c58209c]) (logid:cd7806d0) Seq 2-3926857400090361879: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:38:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62d8b6ad]) (logid:c0b4fca9) No inactive management server node found 2025-06-29 03:38:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62d8b6ad]) (logid:c0b4fca9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:00,077 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e1a7f4b6]) (logid:207f5161) HA health check task is running... 2025-06-29 03:39:01,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17eafc7d]) (logid:573faced) No inactive management server node found 2025-06-29 03:39:01,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17eafc7d]) (logid:573faced) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-634d6211]) (logid:17fa0b1b) No inactive management server node found 2025-06-29 03:39:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-634d6211]) (logid:17fa0b1b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:04,079 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4392b0d3]) (logid:7117d8d3) HA health check task is running... 2025-06-29 03:39:04,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31c3a1fe]) (logid:4876c90f) No inactive management server node found 2025-06-29 03:39:04,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31c3a1fe]) (logid:4876c90f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:39:05,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8260: Seq 7-8260: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:39:05,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8260 2025-06-29 03:39:05,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce6058d2]) (logid:30805eb6) No inactive management server node found 2025-06-29 03:39:05,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce6058d2]) (logid:30805eb6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed6b484d]) (logid:a83bd6ce) No inactive management server node found 2025-06-29 03:39:07,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed6b484d]) (logid:a83bd6ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:08,081 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7030206c]) (logid:412ce057) HA health check task is running... 2025-06-29 03:39:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fed81a15]) (logid:47bf7b46) No inactive management server node found 2025-06-29 03:39:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fed81a15]) (logid:47bf7b46) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0478def7]) (logid:e8f82248) No inactive management server node found 2025-06-29 03:39:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0478def7]) (logid:e8f82248) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb85a6b8]) (logid:9ab770cf) No inactive management server node found 2025-06-29 03:39:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb85a6b8]) (logid:9ab770cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:12,082 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2ebfcf28]) (logid:d89ee766) HA health check task is running... 2025-06-29 03:39:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b568ba37]) (logid:1b0d5c6b) Found 0 routers to update status. 2025-06-29 03:39:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b568ba37]) (logid:1b0d5c6b) Found 0 VPC's to update Redundant State. 2025-06-29 03:39:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b568ba37]) (logid:1b0d5c6b) Found 0 networks to update RvR status. 2025-06-29 03:39:13,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe2bbdb3]) (logid:c6cd4e9a) No inactive management server node found 2025-06-29 03:39:13,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe2bbdb3]) (logid:c6cd4e9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a605fc4]) (logid:67b6be27) No inactive management server node found 2025-06-29 03:39:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a605fc4]) (logid:67b6be27) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:15,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8261: Seq 7-8261: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:39:15,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8261 2025-06-29 03:39:16,083 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c224218e]) (logid:cbc39d15) HA health check task is running... 2025-06-29 03:39:16,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddde7d8d]) (logid:68ece19b) No inactive management server node found 2025-06-29 03:39:16,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddde7d8d]) (logid:68ece19b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:17,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48087846]) (logid:624f920c) No inactive management server node found 2025-06-29 03:39:17,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48087846]) (logid:624f920c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:19,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4273bfe5]) (logid:7ed7eb10) No inactive management server node found 2025-06-29 03:39:19,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4273bfe5]) (logid:7ed7eb10) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:20,085 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-803c0cb9]) (logid:a64a9e31) HA health check task is running... 2025-06-29 03:39:20,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4422cc43]) (logid:ce7881f5) No inactive management server node found 2025-06-29 03:39:20,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4422cc43]) (logid:ce7881f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:22,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0bc918a]) (logid:57f637ca) No inactive management server node found 2025-06-29 03:39:22,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0bc918a]) (logid:57f637ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:23,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81b774e8]) (logid:aab57881) No inactive management server node found 2025-06-29 03:39:23,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81b774e8]) (logid:aab57881) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:24,087 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b923343e]) (logid:5874b2c9) HA health check task is running... 2025-06-29 03:39:25,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8262: Seq 7-8262: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:39:25,225 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8262 2025-06-29 03:39:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cb700bb]) (logid:7f6bd85f) No inactive management server node found 2025-06-29 03:39:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cb700bb]) (logid:7f6bd85f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f796be0d]) (logid:913deaab) No inactive management server node found 2025-06-29 03:39:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f796be0d]) (logid:913deaab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:28,088 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5b4e94d0]) (logid:b49d6920) HA health check task is running... 2025-06-29 03:39:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e62d5823]) (logid:6c7cb8ff) Enabled non-edge zones available for scan: 1 2025-06-29 03:39:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e62d5823]) (logid:6c7cb8ff) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:39:28,100 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e62d5823]) (logid:6c7cb8ff) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:39:28,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3bdcbef3]) (logid:19231a78) Enabled non-edge zones available for scan: 1 2025-06-29 03:39:28,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3bdcbef3]) (logid:19231a78) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:39:28,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51da7a08]) (logid:703fea81) No inactive management server node found 2025-06-29 03:39:28,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51da7a08]) (logid:703fea81) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb6cd1e5]) (logid:fe6e5ad4) No inactive management server node found 2025-06-29 03:39:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb6cd1e5]) (logid:fe6e5ad4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:31,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0165183a]) (logid:56adb622) No inactive management server node found 2025-06-29 03:39:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0165183a]) (logid:56adb622) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:32,089 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f9cd6844]) (logid:8105aa2b) HA health check task is running... 2025-06-29 03:39:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6283c124]) (logid:6f7f9fc9) No inactive management server node found 2025-06-29 03:39:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6283c124]) (logid:6f7f9fc9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:34,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a0b1275]) (logid:dcfb0746) No inactive management server node found 2025-06-29 03:39:34,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a0b1275]) (logid:dcfb0746) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:35,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8263: Seq 7-8263: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:39:35,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8263 2025-06-29 03:39:35,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a037aea]) (logid:b4adafb5) No inactive management server node found 2025-06-29 03:39:35,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a037aea]) (logid:b4adafb5) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:36,091 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3f26f8bc]) (logid:1ce65dd6) HA health check task is running... 2025-06-29 03:39:37,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f543562a]) (logid:0167aa74) No inactive management server node found 2025-06-29 03:39:37,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f543562a]) (logid:0167aa74) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:39:38,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-887e26ee]) (logid:07726f39) No inactive management server node found 2025-06-29 03:39:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-887e26ee]) (logid:07726f39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:40,092 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-fb400505]) (logid:1745186f) HA health check task is running... 2025-06-29 03:39:40,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74bc06b6]) (logid:1f85fcb9) No inactive management server node found 2025-06-29 03:39:40,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74bc06b6]) (logid:1f85fcb9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:41,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1fa9e56]) (logid:93501138) No inactive management server node found 2025-06-29 03:39:41,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1fa9e56]) (logid:93501138) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-bd9d8c29]) (logid:2f261bc5) DbCollector is running... 2025-06-29 03:39:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b7a05608]) (logid:313cd622) Found 0 routers to update status. 2025-06-29 03:39:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b7a05608]) (logid:313cd622) Found 0 VPC's to update Redundant State. 2025-06-29 03:39:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b7a05608]) (logid:313cd622) Found 0 networks to update RvR status. 2025-06-29 03:39:43,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bf15190]) (logid:bf1ec8b4) No inactive management server node found 2025-06-29 03:39:43,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bf15190]) (logid:bf1ec8b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:39:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) ManagementServerCollector is running... 2025-06-29 03:39:44,042 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) DB properties were already loaded 2025-06-29 03:39:44,042 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:39:44,057 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Execution of process [4064923] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:39:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Exit value of process [4064923] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:39:44,057 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Process [4064923] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:39:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) usage install: null 2025-06-29 03:39:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Sessions active: 0 2025-06-29 03:39:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:39:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064926] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:39:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:39:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064931] for command [/bin/bash -c uname -r ]. 2025-06-29 03:39:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:39:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064932] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:39:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) system memory from /proc: 4887515136 2025-06-29 03:39:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:39:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064937] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:39:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) free memory from /proc: 155197440 2025-06-29 03:39:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:39:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064942] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:39:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) used memory from /proc: 1070816 2025-06-29 03:39:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:39:44,094 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-028860b7]) (logid:2e22c1b9) HA health check task is running... 2025-06-29 03:39:44,096 DEBUG [o.a.c.m.w.W.WebhookDeliveryCleanupWorker] (Webhook-Deliveries-Cleanup-Worker-1:[ctx-d810ae3e]) (logid:0b559a95) Clearing old deliveries for webhooks with limit=10 using management server ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:39:44,099 DEBUG [o.a.c.m.w.W.WebhookDeliveryCleanupWorker] (Webhook-Deliveries-Cleanup-Worker-1:[ctx-d810ae3e]) (logid:0b559a95) Cleared old deliveries with limit=10 for 0 webhooks 2025-06-29 03:39:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064943] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:39:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:39:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064944] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:39:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:39:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064945] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:39:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:39:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064952] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:39:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:39:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Successfully executed process [4064953] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:39:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail buffersdirect.capacity, 165609 2025-06-29 03:39:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail buffersdirect.count, 59 2025-06-29 03:39:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail buffersdirect.used, 165609 2025-06-29 03:39:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail buffersmapped.capacity, 0 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail buffersmapped.count, 0 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail buffersmapped.used, 0 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail jvmuptime, 243834 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memoryheap.committed, 517996544 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memoryheap.init, 77594624 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memoryheap.usage, 0.18989707990780993 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorynon-heap.committed, 185597952 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorynon-heap.max, -1 2025-06-29 03:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorynon-heap.usage, -1.82296648E8 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorynon-heap.used, 182296648 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30289473684210527 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.08282374700079978 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.1822085263438531 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012121722102165222 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.Metaspace.usage, 0.99468994140625 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.PS-Eden-Space.usage, 0.27293178714865746 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorytotal.committed, 703594496 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorytotal.init, 85262336 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorytotal.max, 1908932607 2025-06-29 03:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail memorytotal.used, 545184024 2025-06-29 03:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail threadsdeadlocks, [] 2025-06-29 03:39:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail threadsnew.count, 0 2025-06-29 03:39:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) not storing detail threadstimed_waiting.count, 82 2025-06-29 03:39:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-436e0ad7]) (logid:ec4e323c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":156650024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":243760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":543633024,"totalCommitted":703594496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":362500728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":155197440,"systemMemoryUsed":1070816,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.36,0.25],"systemCyclesUsage":[66395419,53820635,4588712896],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:39:44,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 7, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":156650024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":243760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":543633024,"totalCommitted":703594496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":362500728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":155197440,"systemMemoryUsed":1070816,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.36,0.25],"systemCyclesUsage":[66395419,53820635,4588712896],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:39:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:39:44,146 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-7:[ctx-6af0e663]) (logid:ecf95f98) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":156650024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":243760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":543633024,"totalCommitted":703594496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":362500728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":155197440,"systemMemoryUsed":1070816,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.36,0.25],"systemCyclesUsage":[66395419,53820635,4588712896],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:39:44,147 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:39:44,147 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 7, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":156650024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":243760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":543633024,"totalCommitted":703594496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":362500728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":155197440,"systemMemoryUsed":1070816,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.36,0.25],"systemCyclesUsage":[66395419,53820635,4588712896],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:39:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fda81c9e]) (logid:1a19a169) No inactive management server node found 2025-06-29 03:39:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fda81c9e]) (logid:1a19a169) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:45,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8264: Seq 7-8264: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:39:45,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8264 2025-06-29 03:39:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:39:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:39:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a72466f]) (logid:ad65fca7) No inactive management server node found 2025-06-29 03:39:46,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a72466f]) (logid:ad65fca7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:46,647 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:39:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:39:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:39:46,665 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":50,"instanceName":"i-5-50-VM","state":"Running","type":"User","uuid":"1ef8b8a0-f773-4867-b491-342785c57e98"}, power state: PowerOn 2025-06-29 03:39:46,677 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-8:[]) (logid:) VM instance {"id":50,"instanceName":"i-5-50-VM","state":"Running","type":"User","uuid":"1ef8b8a0-f773-4867-b491-342785c57e98"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:39:46,678 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":74,"instanceName":"i-5-74-VM","state":"Running","type":"User","uuid":"01b29297-2d6d-4fe9-9006-7c66f30d5043"}, power state: PowerOn 2025-06-29 03:39:46,689 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-8:[]) (logid:) VM instance {"id":74,"instanceName":"i-5-74-VM","state":"Running","type":"User","uuid":"01b29297-2d6d-4fe9-9006-7c66f30d5043"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:39:46,691 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) VM state report is updated. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"}, power state: PowerOn 2025-06-29 03:39:46,702 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-8:[]) (logid:) VM instance {"id":303,"instanceName":"i-5-303-VM","state":"Running","type":"User","uuid":"811ca2c7-30d0-49d1-956d-8e5d6cda2e8e"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, new host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, host before state transition: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:39:46,703 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:39:47,075 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:39:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":2,"instanceName":"s-2-VM","state":"Running","type":"SecondaryStorageVm","uuid":"9e5a7cce-ef20-426e-b6fe-c076abf53264"}, power state: PowerOn 2025-06-29 03:39:47,104 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-9:[]) (logid:) VM instance {"id":2,"instanceName":"s-2-VM","state":"Running","type":"SecondaryStorageVm","uuid":"9e5a7cce-ef20-426e-b6fe-c076abf53264"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,106 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":306,"instanceName":"v-306-VM","state":"Running","type":"ConsoleProxy","uuid":"d1ec8241-db64-4a9e-8f6f-866239a02a33"}, power state: PowerOn 2025-06-29 03:39:47,118 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-9:[]) (logid:) VM instance {"id":306,"instanceName":"v-306-VM","state":"Running","type":"ConsoleProxy","uuid":"d1ec8241-db64-4a9e-8f6f-866239a02a33"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,120 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":110,"instanceName":"i-5-110-VM","state":"Running","type":"User","uuid":"fd1af66f-e131-42ef-9f31-42f4a54405d7"}, power state: PowerOn 2025-06-29 03:39:47,130 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-9:[]) (logid:) VM instance {"id":110,"instanceName":"i-5-110-VM","state":"Running","type":"User","uuid":"fd1af66f-e131-42ef-9f31-42f4a54405d7"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,132 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"}, power state: PowerOn 2025-06-29 03:39:47,141 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-9:[]) (logid:) VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,143 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) VM state report is updated. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, VM instance {"id":127,"instanceName":"i-5-127-VM","state":"Running","type":"User","uuid":"3fa040d9-feea-484c-aba4-3a7770ff6209"}, power state: PowerOn 2025-06-29 03:39:47,154 DEBUG [c.c.c.CapacityManagerImpl] (AgentManager-Handler-9:[]) (logid:) VM instance {"id":127,"instanceName":"i-5-127-VM","state":"Running","type":"User","uuid":"3fa040d9-feea-484c-aba4-3a7770ff6209"} state transited from [Running] to [Running] with event [FollowAgentPowerOnReport]. VM's original host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, new host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, host before state transition: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,155 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:39:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d531d94]) (logid:816f64e7) No inactive management server node found 2025-06-29 03:39:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d531d94]) (logid:816f64e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:48,066 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ecdc83e6]) (logid:e9f61014) ClusterDRS.poll is being called at 2025-06-29T01:40:00+0000 2025-06-29 03:39:48,076 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ecdc83e6]) (logid:e9f61014) Removed 0 old drs migration plans 2025-06-29 03:39:48,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-64f2f57c]) (logid:1a06265b) Resetting hosts suitable for reconnect 2025-06-29 03:39:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-64f2f57c]) (logid:1a06265b) Completed resetting hosts suitable for reconnect 2025-06-29 03:39:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-64f2f57c]) (logid:1a06265b) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:39:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-64f2f57c]) (logid:1a06265b) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:39:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-64f2f57c]) (logid:1a06265b) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:39:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-64f2f57c]) (logid:1a06265b) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:39:48,095 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2dc031fb]) (logid:f1c3d2b6) HA health check task is running... 2025-06-29 03:39:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7ccbae58]) (logid:c2d73717) VM scheduler.poll is being called at 2025-06-29T01:40:00+0000 2025-06-29 03:39:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7ccbae58]) (logid:c2d73717) Got 0 scheduled jobs to be executed at 2025-06-29T01:40:00+0000 2025-06-29 03:39:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7ccbae58]) (logid:c2d73717) Cleaned up 0 VM scheduled job entries 2025-06-29 03:39:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-2d91a48f]) (logid:97e319fc) VM Operation Thread Running 2025-06-29 03:39:49,377 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d68ee80]) (logid:41d6ab93) No inactive management server node found 2025-06-29 03:39:49,378 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d68ee80]) (logid:41d6ab93) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 5ms 2025-06-29 03:39:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-430acf7b]) (logid:1941b800) No inactive management server node found 2025-06-29 03:39:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-430acf7b]) (logid:1941b800) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:52,096 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-63e5a2cd]) (logid:acb5b74f) HA health check task is running... 2025-06-29 03:39:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-203c8742]) (logid:c1914c08) No inactive management server node found 2025-06-29 03:39:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-203c8742]) (logid:c1914c08) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18a5e085]) (logid:9a459738) No inactive management server node found 2025-06-29 03:39:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18a5e085]) (logid:9a459738) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:55,218 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8266: Seq 7-8266: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:39:55,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8266 2025-06-29 03:39:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84df88f0]) (logid:97564757) No inactive management server node found 2025-06-29 03:39:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84df88f0]) (logid:97564757) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:56,098 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c0dd1663]) (logid:96b707ae) HA health check task is running... 2025-06-29 03:39:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc40ed3]) (logid:3ab7f57f) No inactive management server node found 2025-06-29 03:39:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc40ed3]) (logid:3ab7f57f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:58,052 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2994d53c]) (logid:44708d91) Starting to remove all volume_stats rows older than [Sat Jun 28 15:39:58 CEST 2025]. 2025-06-29 03:39:58,053 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2994d53c]) (logid:44708d91) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:39:58 CEST 2025]. 2025-06-29 03:39:58,059 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-ef2b2651]) (logid:67f83f7e) AutoScaling Monitor is running... 2025-06-29 03:39:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-28a1a5ff]) (logid:66142897) Enabled non-edge zones available for scan: 1 2025-06-29 03:39:58,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-28a1a5ff]) (logid:66142897) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:39:58,100 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-28a1a5ff]) (logid:66142897) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:39:58,126 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:39:58,128 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:39:58,129 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) Seq 2-4725683383995203609: Routed from 90520738888109 2025-06-29 03:39:58,133 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) Seq 1-4725683383995203609: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:39:58,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:39:58,135 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) Seq 2-3926857400090361880: Routed from 90520738888109 2025-06-29 03:39:58,137 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a22df88e]) (logid:edfe2085) Starting to remove all vm_stats rows older than [Sat Jun 28 15:39:58 CEST 2025]. 2025-06-29 03:39:58,141 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3fedcd15]) (logid:5e37e683) Seq 2-3926857400090361880: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:39:58,161 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a22df88e]) (logid:edfe2085) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:39:58 CEST 2025]. 2025-06-29 03:39:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f910c41]) (logid:189cf456) Enabled non-edge zones available for scan: 1 2025-06-29 03:39:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7f910c41]) (logid:189cf456) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:39:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44892105]) (logid:e8978e80) No inactive management server node found 2025-06-29 03:39:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44892105]) (logid:e8978e80) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:39:58,956 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:39:58,961 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:39:58,962 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) Seq 2-4725683383995203610: Routed from 90520738888109 2025-06-29 03:39:59,040 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) Seq 1-4725683383995203610: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:39:59,075 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:39:59,076 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) Seq 2-3926857400090361881: Routed from 90520738888109 2025-06-29 03:39:59,080 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) StorageCollector is running... 2025-06-29 03:39:59,088 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:39:59,088 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:39:59,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:39:59,089 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-3938116399158788107: Routed from 90520738888109 2025-06-29 03:39:59,154 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 4-3938116399158788107: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:39:59,160 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:39:59,160 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:39:59,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:39:59,161 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-3926857400090361882: Routed from 90520738888109 2025-06-29 03:39:59,197 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-3926857400090361882: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:39:59,204 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bc018dab]) (logid:1e6012dc) Seq 2-3926857400090361881: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:39:59,209 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:39:59,209 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:39:59,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:39:59,211 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-4725683383995203611: Routed from 90520738888109 2025-06-29 03:39:59,260 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 1-4725683383995203611: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:39:59,275 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:39:59,275 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:39:59,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:39:59,277 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-4725683383995203612: Routed from 90520738888109 2025-06-29 03:39:59,363 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 1-4725683383995203612: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:39:59,366 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:39:59,366 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:39:59,366 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:39:59,368 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-3926857400090361883: Routed from 90520738888109 2025-06-29 03:39:59,413 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5e8a7c54]) (logid:7769733f) Seq 2-3926857400090361883: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:39:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b94c737d]) (logid:8a76591d) No inactive management server node found 2025-06-29 03:39:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b94c737d]) (logid:8a76591d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:00,099 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e428fc09]) (logid:fe671198) HA health check task is running... 2025-06-29 03:40:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-735fb21b]) (logid:b995cd28) No inactive management server node found 2025-06-29 03:40:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-735fb21b]) (logid:b995cd28) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:02,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c48a6cd]) (logid:833963ee) No inactive management server node found 2025-06-29 03:40:02,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c48a6cd]) (logid:833963ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:04,100 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a9ebce18]) (logid:011758b5) HA health check task is running... 2025-06-29 03:40:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dadf89e]) (logid:0de07252) No inactive management server node found 2025-06-29 03:40:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dadf89e]) (logid:0de07252) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:05,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8267: Seq 7-8267: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:40:05,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8267 2025-06-29 03:40:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d400fb28]) (logid:b0a7c1b7) No inactive management server node found 2025-06-29 03:40:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d400fb28]) (logid:b0a7c1b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:07,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c486e811]) (logid:a617dd7c) No inactive management server node found 2025-06-29 03:40:07,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c486e811]) (logid:a617dd7c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:08,102 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d211788d]) (logid:938bf425) HA health check task is running... 2025-06-29 03:40:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a4c4d50]) (logid:b728fd56) No inactive management server node found 2025-06-29 03:40:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a4c4d50]) (logid:b728fd56) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:10,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98d91d3d]) (logid:dad951a4) No inactive management server node found 2025-06-29 03:40:10,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98d91d3d]) (logid:dad951a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:11,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cd719f0]) (logid:ccff6ea9) No inactive management server node found 2025-06-29 03:40:11,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cd719f0]) (logid:ccff6ea9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:12,104 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ac83b500]) (logid:078a42ab) HA health check task is running... 2025-06-29 03:40:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2622fcb]) (logid:c212b2e5) Found 0 routers to update status. 2025-06-29 03:40:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2622fcb]) (logid:c212b2e5) Found 0 VPC's to update Redundant State. 2025-06-29 03:40:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2622fcb]) (logid:c212b2e5) Found 0 networks to update RvR status. 2025-06-29 03:40:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd3c70c2]) (logid:0fc25e70) No inactive management server node found 2025-06-29 03:40:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd3c70c2]) (logid:0fc25e70) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60b5745b]) (logid:30fefca2) No inactive management server node found 2025-06-29 03:40:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60b5745b]) (logid:30fefca2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:15,218 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8268: Seq 7-8268: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:40:15,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8268 2025-06-29 03:40:16,105 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-174a009c]) (logid:91d7e7b5) HA health check task is running... 2025-06-29 03:40:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-246df18c]) (logid:1474f8fa) No inactive management server node found 2025-06-29 03:40:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-246df18c]) (logid:1474f8fa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:17,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e01830c6]) (logid:fc3b4a27) No inactive management server node found 2025-06-29 03:40:17,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e01830c6]) (logid:fc3b4a27) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c41996f4]) (logid:9184a2d1) No inactive management server node found 2025-06-29 03:40:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c41996f4]) (logid:9184a2d1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:20,107 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e507ad6d]) (logid:615ca208) HA health check task is running... 2025-06-29 03:40:20,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfeee572]) (logid:f9ec3094) No inactive management server node found 2025-06-29 03:40:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfeee572]) (logid:f9ec3094) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f49ed5e1]) (logid:52e7b8f5) No inactive management server node found 2025-06-29 03:40:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f49ed5e1]) (logid:52e7b8f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41ede6ad]) (logid:355a5ad9) No inactive management server node found 2025-06-29 03:40:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41ede6ad]) (logid:355a5ad9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:24,108 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-46ad23ad]) (logid:4e21fbb5) HA health check task is running... 2025-06-29 03:40:25,218 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8269: Seq 7-8269: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:40:25,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8269 2025-06-29 03:40:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08713c94]) (logid:18dff4aa) No inactive management server node found 2025-06-29 03:40:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08713c94]) (logid:18dff4aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c242f00b]) (logid:6297496e) No inactive management server node found 2025-06-29 03:40:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c242f00b]) (logid:6297496e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-37c90b0c]) (logid:3ff7fc7c) Enabled non-edge zones available for scan: 1 2025-06-29 03:40:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-37c90b0c]) (logid:3ff7fc7c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:40:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-37c90b0c]) (logid:3ff7fc7c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:40:28,109 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5d325d72]) (logid:4f61f947) HA health check task is running... 2025-06-29 03:40:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b222ae61]) (logid:b346358e) Enabled non-edge zones available for scan: 1 2025-06-29 03:40:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b222ae61]) (logid:b346358e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:40:28,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71cb4f47]) (logid:d7c73f4e) No inactive management server node found 2025-06-29 03:40:28,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71cb4f47]) (logid:d7c73f4e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:29,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e32af7a]) (logid:15a0f8ad) No inactive management server node found 2025-06-29 03:40:29,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e32af7a]) (logid:15a0f8ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:30,219 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8270: Seq 7-8270: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:40:30,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8270 2025-06-29 03:40:31,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3609b952]) (logid:ac9efc9e) No inactive management server node found 2025-06-29 03:40:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3609b952]) (logid:ac9efc9e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:32,110 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ea7aee02]) (logid:f82a15b3) HA health check task is running... 2025-06-29 03:40:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd47dd0c]) (logid:82b81ee7) No inactive management server node found 2025-06-29 03:40:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd47dd0c]) (logid:82b81ee7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3db9753c]) (logid:0af39a94) No inactive management server node found 2025-06-29 03:40:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3db9753c]) (logid:0af39a94) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-849489e4]) (logid:78e32a6d) No inactive management server node found 2025-06-29 03:40:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-849489e4]) (logid:78e32a6d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:36,112 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d94a0b71]) (logid:4f55e511) HA health check task is running... 2025-06-29 03:40:37,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ee98062]) (logid:eddff3f4) No inactive management server node found 2025-06-29 03:40:37,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ee98062]) (logid:eddff3f4) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms 2025-06-29 03:40:38,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34e9a2d5]) (logid:e7d0a9f2) No inactive management server node found 2025-06-29 03:40:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34e9a2d5]) (logid:e7d0a9f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:40,114 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8b84e3c4]) (logid:331722a5) HA health check task is running... 2025-06-29 03:40:40,219 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8271: Seq 7-8271: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:40:40,229 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8271 2025-06-29 03:40:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-366f9732]) (logid:dca20378) No inactive management server node found 2025-06-29 03:40:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-366f9732]) (logid:dca20378) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:41,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1cef1b1]) (logid:1a504144) No inactive management server node found 2025-06-29 03:40:41,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1cef1b1]) (logid:1a504144) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-e98526a2]) (logid:7af7dcdf) DbCollector is running... 2025-06-29 03:40:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3c9170bb]) (logid:28dbd24f) Found 0 routers to update status. 2025-06-29 03:40:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3c9170bb]) (logid:28dbd24f) Found 0 VPC's to update Redundant State. 2025-06-29 03:40:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3c9170bb]) (logid:28dbd24f) Found 0 networks to update RvR status. 2025-06-29 03:40:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a7bd8dd]) (logid:bdcd2665) No inactive management server node found 2025-06-29 03:40:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a7bd8dd]) (logid:bdcd2665) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) ManagementServerCollector is running... 2025-06-29 03:40:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) DB properties were already loaded 2025-06-29 03:40:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:40:44,060 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Execution of process [4064967] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:40:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Exit value of process [4064967] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:40:44,060 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Process [4064967] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:40:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) usage install: null 2025-06-29 03:40:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Sessions active: 0 2025-06-29 03:40:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:40:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064970] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:40:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:40:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064975] for command [/bin/bash -c uname -r ]. 2025-06-29 03:40:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:40:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064976] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:40:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) system memory from /proc: 4887515136 2025-06-29 03:40:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:40:44,085 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064981] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:40:44,086 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) free memory from /proc: 158339072 2025-06-29 03:40:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:40:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064986] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:40:44,098 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) used memory from /proc: 1071200 2025-06-29 03:40:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:40:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064987] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:40:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:40:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064988] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:40:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:40:44,115 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c40ace73]) (logid:b801c255) HA health check task is running... 2025-06-29 03:40:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064989] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:40:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:40:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064996] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:40:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:40:44,132 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Successfully executed process [4064997] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:40:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail buffersdirect.capacity, 183998 2025-06-29 03:40:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail buffersdirect.count, 60 2025-06-29 03:40:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail buffersdirect.used, 183998 2025-06-29 03:40:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail buffersmapped.capacity, 0 2025-06-29 03:40:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail buffersmapped.count, 0 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail buffersmapped.used, 0 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail jvmuptime, 303838 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memoryheap.committed, 517996544 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memoryheap.init, 77594624 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memoryheap.usage, 0.20662087197161022 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorynon-heap.committed, 185860096 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorynon-heap.max, -1 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorynon-heap.usage, -1.82954424E8 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorynon-heap.used, 182954424 2025-06-29 03:40:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30289473684210527 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.08465867435350573 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.18499016896057588 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.Compressed-Class-Space.usage, 0.012124806642532349 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.Metaspace.usage, 0.9948424556185116 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.PS-Eden-Space.usage, 0.32191919033698085 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorytotal.committed, 703856640 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorytotal.init, 85262336 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorytotal.max, 1908932607 2025-06-29 03:40:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail memorytotal.used, 577766784 2025-06-29 03:40:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail threadsdeadlocks, [] 2025-06-29 03:40:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail threadsnew.count, 0 2025-06-29 03:40:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:40:44,141 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-8d113907]) (logid:fc18ed57) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":124340720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":303764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":576592424,"totalCommitted":703856640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":394425320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":158339072,"systemMemoryUsed":1071200,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.29,0.23],"systemCyclesUsage":[66395859,53820952,4588735892],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:40:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 9, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":124340720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":303764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":576592424,"totalCommitted":703856640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":394425320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":158339072,"systemMemoryUsed":1071200,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.29,0.23],"systemCyclesUsage":[66395859,53820952,4588735892],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:40:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:40:44,151 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:40:44,151 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 9, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":124340720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":303764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":576592424,"totalCommitted":703856640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":394425320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":158339072,"systemMemoryUsed":1071200,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.29,0.23],"systemCyclesUsage":[66395859,53820952,4588735892],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:40:44,151 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-8:[ctx-4540e91f]) (logid:d29d3484) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":124340720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":303764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":576592424,"totalCommitted":703856640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":394425320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":158339072,"systemMemoryUsed":1071200,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.29,0.23],"systemCyclesUsage":[66395859,53820952,4588735892],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:40:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34c6d14a]) (logid:8e010365) No inactive management server node found 2025-06-29 03:40:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34c6d14a]) (logid:8e010365) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:40:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:40:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71396f92]) (logid:71c0df0b) No inactive management server node found 2025-06-29 03:40:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71396f92]) (logid:71c0df0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:40:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:40:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:40:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:40:47,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:40:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:40:47,074 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:40:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:40:47,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c90c0f3a]) (logid:5dd3d376) No inactive management server node found 2025-06-29 03:40:47,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c90c0f3a]) (logid:5dd3d376) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-05be3a49]) (logid:2fd71f8b) ClusterDRS.poll is being called at 2025-06-29T01:41:00+0000 2025-06-29 03:40:48,073 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-05be3a49]) (logid:2fd71f8b) Removed 0 old drs migration plans 2025-06-29 03:40:48,117 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7d06f85d]) (logid:e0c1c81d) HA health check task is running... 2025-06-29 03:40:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-90f0a8b8]) (logid:55ba541e) VM scheduler.poll is being called at 2025-06-29T01:41:00+0000 2025-06-29 03:40:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-90f0a8b8]) (logid:55ba541e) Got 0 scheduled jobs to be executed at 2025-06-29T01:41:00+0000 2025-06-29 03:40:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-90f0a8b8]) (logid:55ba541e) Cleaned up 0 VM scheduled job entries 2025-06-29 03:40:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-f640d791]) (logid:b7c13d74) VM Operation Thread Running 2025-06-29 03:40:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ada2c8d5]) (logid:6fa306b8) No inactive management server node found 2025-06-29 03:40:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ada2c8d5]) (logid:6fa306b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:50,219 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8273: Seq 7-8273: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:40:50,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8273 2025-06-29 03:40:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f314a686]) (logid:de9b2161) No inactive management server node found 2025-06-29 03:40:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f314a686]) (logid:de9b2161) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:52,118 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0b37225e]) (logid:37bc6612) HA health check task is running... 2025-06-29 03:40:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07fd12ed]) (logid:cbc711a7) No inactive management server node found 2025-06-29 03:40:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07fd12ed]) (logid:cbc711a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa70d688]) (logid:e3fc7fcc) No inactive management server node found 2025-06-29 03:40:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa70d688]) (logid:e3fc7fcc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:55,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63a0967b]) (logid:e26df294) No inactive management server node found 2025-06-29 03:40:55,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63a0967b]) (logid:e26df294) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:56,120 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fd5d69c9]) (logid:8c610bfe) HA health check task is running... 2025-06-29 03:40:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba87e6fa]) (logid:0fe6005b) No inactive management server node found 2025-06-29 03:40:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba87e6fa]) (logid:0fe6005b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:58,055 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-b2e65bec]) (logid:f2d4c8a9) Starting to remove all volume_stats rows older than [Sat Jun 28 15:40:58 CEST 2025]. 2025-06-29 03:40:58,056 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-b2e65bec]) (logid:f2d4c8a9) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:40:58 CEST 2025]. 2025-06-29 03:40:58,061 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-5b79dabd]) (logid:53948fdf) AutoScaling Monitor is running... 2025-06-29 03:40:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-65e9f67b]) (logid:c5533161) Enabled non-edge zones available for scan: 1 2025-06-29 03:40:58,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-65e9f67b]) (logid:c5533161) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:40:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-65e9f67b]) (logid:c5533161) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:40:58,145 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:40:58,147 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:40:58,149 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) Seq 2-4725683383995203613: Routed from 90520738888109 2025-06-29 03:40:58,164 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6242c044]) (logid:f63c5037) Starting to remove all vm_stats rows older than [Sat Jun 28 15:40:58 CEST 2025]. 2025-06-29 03:40:58,189 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6242c044]) (logid:f63c5037) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:40:58 CEST 2025]. 2025-06-29 03:40:58,193 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) Seq 1-4725683383995203613: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:40:58,195 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:40:58,196 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) Seq 2-3926857400090361884: Routed from 90520738888109 2025-06-29 03:40:58,241 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-6f85c135]) (logid:1641deb2) Seq 2-3926857400090361884: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:40:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-535d186c]) (logid:9b460c1e) Enabled non-edge zones available for scan: 1 2025-06-29 03:40:58,290 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-535d186c]) (logid:9b460c1e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:40:58,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cc0d73e]) (logid:146e5b95) No inactive management server node found 2025-06-29 03:40:58,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cc0d73e]) (logid:146e5b95) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:40:59,270 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:40:59,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:40:59,277 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) Seq 2-4725683383995203614: Routed from 90520738888109 2025-06-29 03:40:59,335 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) Seq 1-4725683383995203614: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:40:59,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:40:59,377 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) Seq 2-3926857400090361885: Routed from 90520738888109 2025-06-29 03:40:59,414 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) StorageCollector is running... 2025-06-29 03:40:59,423 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:40:59,423 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:40:59,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:40:59,425 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 2-3938116399158788108: Routed from 90520738888109 2025-06-29 03:40:59,486 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 4-3938116399158788108: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:40:59,493 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:40:59,493 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:40:59,493 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:40:59,495 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 2-4725683383995203615: Routed from 90520738888109 2025-06-29 03:40:59,522 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ecd7488]) (logid:93d0b7d0) Seq 2-3926857400090361885: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:40:59,544 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 1-4725683383995203615: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:40:59,558 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:40:59,558 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:40:59,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:40:59,559 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 2-4725683383995203616: Routed from 90520738888109 2025-06-29 03:40:59,649 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 1-4725683383995203616: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:40:59,660 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:40:59,660 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:40:59,660 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:40:59,662 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 2-4725683383995203617: Routed from 90520738888109 2025-06-29 03:40:59,748 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 1-4725683383995203617: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:40:59,751 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:40:59,751 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:40:59,751 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:40:59,752 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 2-3926857400090361886: Routed from 90520738888109 2025-06-29 03:40:59,756 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-28ebcfb9]) (logid:6e671c81) Seq 2-3926857400090361886: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:40:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80a39394]) (logid:de4074f8) No inactive management server node found 2025-06-29 03:40:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80a39394]) (logid:de4074f8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:00,122 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-33ad1e4d]) (logid:678fc579) HA health check task is running... 2025-06-29 03:41:00,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8274: Seq 7-8274: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:00,241 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8274 2025-06-29 03:41:01,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8695639]) (logid:2bce1cd2) No inactive management server node found 2025-06-29 03:41:01,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8695639]) (logid:2bce1cd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:02,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-257d8992]) (logid:7833df14) No inactive management server node found 2025-06-29 03:41:02,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-257d8992]) (logid:7833df14) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:04,123 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-486e3301]) (logid:4d8c8a0a) HA health check task is running... 2025-06-29 03:41:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd5759d0]) (logid:c7203bb6) No inactive management server node found 2025-06-29 03:41:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd5759d0]) (logid:c7203bb6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85999e90]) (logid:473fa4b1) No inactive management server node found 2025-06-29 03:41:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85999e90]) (logid:473fa4b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:07,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da257d02]) (logid:b92a2794) No inactive management server node found 2025-06-29 03:41:07,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da257d02]) (logid:b92a2794) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:41:08,125 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5497c492]) (logid:5aa7ab0c) HA health check task is running... 2025-06-29 03:41:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df62dd41]) (logid:e8d35b65) No inactive management server node found 2025-06-29 03:41:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df62dd41]) (logid:e8d35b65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:10,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8275: Seq 7-8275: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:10,230 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8275 2025-06-29 03:41:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1096ba9d]) (logid:e25cb98c) No inactive management server node found 2025-06-29 03:41:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1096ba9d]) (logid:e25cb98c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:11,867 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecb6d559]) (logid:af7031c6) No inactive management server node found 2025-06-29 03:41:11,867 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecb6d559]) (logid:af7031c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:12,126 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-fddc46c7]) (logid:b75af410) HA health check task is running... 2025-06-29 03:41:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-10131d9d]) (logid:2177971b) Found 0 routers to update status. 2025-06-29 03:41:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-10131d9d]) (logid:2177971b) Found 0 VPC's to update Redundant State. 2025-06-29 03:41:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-10131d9d]) (logid:2177971b) Found 0 networks to update RvR status. 2025-06-29 03:41:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4616974b]) (logid:7f33c195) No inactive management server node found 2025-06-29 03:41:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4616974b]) (logid:7f33c195) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b9f3e9b]) (logid:746c16e4) No inactive management server node found 2025-06-29 03:41:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b9f3e9b]) (logid:746c16e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:16,128 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ff316773]) (logid:86da2439) HA health check task is running... 2025-06-29 03:41:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbd3352e]) (logid:28f5a1f5) No inactive management server node found 2025-06-29 03:41:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbd3352e]) (logid:28f5a1f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e3c0603]) (logid:37369532) No inactive management server node found 2025-06-29 03:41:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e3c0603]) (logid:37369532) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:18,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-180e5f39]) (logid:3aaa82a2) Resetting hosts suitable for reconnect 2025-06-29 03:41:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-180e5f39]) (logid:3aaa82a2) Completed resetting hosts suitable for reconnect 2025-06-29 03:41:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-180e5f39]) (logid:3aaa82a2) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:41:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-180e5f39]) (logid:3aaa82a2) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:41:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-180e5f39]) (logid:3aaa82a2) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:41:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-180e5f39]) (logid:3aaa82a2) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:41:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14d6dd9b]) (logid:44579e78) No inactive management server node found 2025-06-29 03:41:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14d6dd9b]) (logid:44579e78) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:20,129 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5f2750ba]) (logid:e80773da) HA health check task is running... 2025-06-29 03:41:20,221 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8276: Seq 7-8276: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:20,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8276 2025-06-29 03:41:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4da35aca]) (logid:6786062c) No inactive management server node found 2025-06-29 03:41:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4da35aca]) (logid:6786062c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-470a6524]) (logid:9daddf75) No inactive management server node found 2025-06-29 03:41:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-470a6524]) (logid:9daddf75) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1b3da12]) (logid:1bd8821c) No inactive management server node found 2025-06-29 03:41:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1b3da12]) (logid:1bd8821c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:24,130 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-16fd1ac2]) (logid:817eb6c0) HA health check task is running... 2025-06-29 03:41:25,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cc771ec]) (logid:d7f5dc9b) No inactive management server node found 2025-06-29 03:41:25,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cc771ec]) (logid:d7f5dc9b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dc831fc]) (logid:2fdbc192) No inactive management server node found 2025-06-29 03:41:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dc831fc]) (logid:2fdbc192) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-87e7680a]) (logid:cdae5338) Enabled non-edge zones available for scan: 1 2025-06-29 03:41:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-87e7680a]) (logid:cdae5338) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:41:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-87e7680a]) (logid:cdae5338) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:41:28,132 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-283b8ae4]) (logid:1673de87) HA health check task is running... 2025-06-29 03:41:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-34cd7b5c]) (logid:ef9b968c) Enabled non-edge zones available for scan: 1 2025-06-29 03:41:28,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-34cd7b5c]) (logid:ef9b968c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:41:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b06fec2d]) (logid:13f5c9db) No inactive management server node found 2025-06-29 03:41:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b06fec2d]) (logid:13f5c9db) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74e42fe]) (logid:eda0c617) No inactive management server node found 2025-06-29 03:41:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74e42fe]) (logid:eda0c617) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:30,221 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8277: Seq 7-8277: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:30,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8277 2025-06-29 03:41:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8301b188]) (logid:f5740720) No inactive management server node found 2025-06-29 03:41:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8301b188]) (logid:f5740720) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:32,134 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-efc1d169]) (logid:cd8322e6) HA health check task is running... 2025-06-29 03:41:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85ecf979]) (logid:1ed7c276) No inactive management server node found 2025-06-29 03:41:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85ecf979]) (logid:1ed7c276) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9b7fad8]) (logid:6e4ba8c4) No inactive management server node found 2025-06-29 03:41:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9b7fad8]) (logid:6e4ba8c4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:35,221 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8278: Seq 7-8278: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:35,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8278 2025-06-29 03:41:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-157309ee]) (logid:b7dd5c81) No inactive management server node found 2025-06-29 03:41:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-157309ee]) (logid:b7dd5c81) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:36,135 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a0d67e6f]) (logid:9871e1c5) HA health check task is running... 2025-06-29 03:41:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3add5dab]) (logid:ae61bc8f) No inactive management server node found 2025-06-29 03:41:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3add5dab]) (logid:ae61bc8f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:38,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b27714d0]) (logid:8aaa2a4d) No inactive management server node found 2025-06-29 03:41:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b27714d0]) (logid:8aaa2a4d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:40,137 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-0ffddc53]) (logid:979fad92) HA health check task is running... 2025-06-29 03:41:40,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-947c4e5b]) (logid:14c7bca6) No inactive management server node found 2025-06-29 03:41:40,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-947c4e5b]) (logid:14c7bca6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:41,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-075dbdcc]) (logid:70eef3a7) No inactive management server node found 2025-06-29 03:41:41,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-075dbdcc]) (logid:70eef3a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-d3a0e840]) (logid:5bfe5788) DbCollector is running... 2025-06-29 03:41:43,093 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-ed1f11db]) (logid:788d53b4) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:41:43,094 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-ed1f11db]) (logid:788d53b4) Snapshot scheduler is being called at [2025-06-29T01:41:43+0000]. 2025-06-29 03:41:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-ed1f11db]) (logid:788d53b4) There are [0] scheduled snapshots to be executed at [2025-06-29T01:41:43+0000]. 2025-06-29 03:41:43,099 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-24f68052]) (logid:ad2b5f34) Found 0 running routers. 2025-06-29 03:41:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b54ead4a]) (logid:88b9264c) Found 0 routers to update status. 2025-06-29 03:41:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b54ead4a]) (logid:88b9264c) Found 0 VPC's to update Redundant State. 2025-06-29 03:41:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b54ead4a]) (logid:88b9264c) Found 0 networks to update RvR status. 2025-06-29 03:41:43,118 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-272b87d9]) (logid:6bf7f84d) External devices are not used. Skipping external device usage collection 2025-06-29 03:41:43,123 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-96169a0b]) (logid:7d32898c) Started resource counters recalculation periodic task. 2025-06-29 03:41:43,127 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-96169a0b]) (logid:7d32898c) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:41:43,304 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-b5fb09ae]) (logid:1e225f6d) Backup backup.poll is being called at 2025-06-29T01:41:43+0000 2025-06-29 03:41:43,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c507f920]) (logid:14f96f3a) No inactive management server node found 2025-06-29 03:41:43,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c507f920]) (logid:14f96f3a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:41:43,677 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-96169a0b]) (logid:7d32898c) Finished resource counters recalculation periodic task. 2025-06-29 03:41:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) ManagementServerCollector is running... 2025-06-29 03:41:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) DB properties were already loaded 2025-06-29 03:41:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:41:44,057 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Execution of process [4065003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:41:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Exit value of process [4065003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:41:44,058 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Process [4065003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:41:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) usage install: null 2025-06-29 03:41:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Sessions active: 0 2025-06-29 03:41:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:41:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065006] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:41:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:41:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065011] for command [/bin/bash -c uname -r ]. 2025-06-29 03:41:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:41:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065012] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:41:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) system memory from /proc: 4887515136 2025-06-29 03:41:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:41:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065017] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:41:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) free memory from /proc: 155529216 2025-06-29 03:41:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:41:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065022] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:41:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) used memory from /proc: 1071840 2025-06-29 03:41:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:41:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065023] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:41:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:41:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065024] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:41:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:41:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065025] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:41:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:41:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065032] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:41:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:41:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Successfully executed process [4065033] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail buffersdirect.capacity, 206338 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail buffersdirect.count, 68 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail buffersdirect.used, 206338 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail buffersmapped.capacity, 0 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail buffersmapped.count, 0 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail buffersmapped.used, 0 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail jvmuptime, 363835 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memoryheap.committed, 517996544 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memoryheap.init, 77594624 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memoryheap.usage, 0.22579630427686634 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorynon-heap.committed, 186515456 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorynon-heap.max, -1 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorynon-heap.usage, -1.83812072E8 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorynon-heap.used, 183812072 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3031798245614035 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.08658211976806185 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.1880571616622788 2025-06-29 03:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.Compressed-Class-Space.usage, 0.012148536741733551 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.Metaspace.usage, 0.9940423101818401 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.PS-Eden-Space.usage, 0.37749756792333944 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorytotal.committed, 704512000 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorytotal.init, 85262336 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorytotal.max, 1908932607 2025-06-29 03:41:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail memorytotal.used, 614845752 2025-06-29 03:41:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail threadsdeadlocks, [] 2025-06-29 03:41:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail threadsnew.count, 0 2025-06-29 03:41:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:41:44,138 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-60a768a9]) (logid:e107a880) HA health check task is running... 2025-06-29 03:41:44,139 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-6e345121]) (logid:cf965ecc) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":87831536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":363761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":613965072,"totalCommitted":704512000,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431029928,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":155529216,"systemMemoryUsed":1071840,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.25,0.22],"systemCyclesUsage":[66396345,53821273,4588758834],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:41:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 11, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":87831536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":363761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":613965072,"totalCommitted":704512000,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431029928,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":155529216,"systemMemoryUsed":1071840,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.25,0.22],"systemCyclesUsage":[66396345,53821273,4588758834],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:41:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:41:44,150 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-8:[ctx-29627c2d]) (logid:4595c3e4) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":87831536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":363761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":613965072,"totalCommitted":704512000,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431029928,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":155529216,"systemMemoryUsed":1071840,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.25,0.22],"systemCyclesUsage":[66396345,53821273,4588758834],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:41:44,150 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 9 ms 2025-06-29 03:41:44,150 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 10ms. agent: 0, pdu seq: 11, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":87831536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":363761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.07,"totalInit":85262336,"totalUsed":613965072,"totalCommitted":704512000,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431029928,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":193,"threadsWaitingCount":101,"systemMemoryTotal":4887515136,"systemMemoryFree":155529216,"systemMemoryUsed":1071840,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.07,0.25,0.22],"systemCyclesUsage":[66396345,53821273,4588758834],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:41:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6de80105]) (logid:8d4ad0c1) No inactive management server node found 2025-06-29 03:41:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6de80105]) (logid:8d4ad0c1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:45,222 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8279: Seq 7-8279: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:45,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8279 2025-06-29 03:41:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:41:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:41:46,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a545b5f7]) (logid:3c200ab3) No inactive management server node found 2025-06-29 03:41:46,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a545b5f7]) (logid:3c200ab3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:46,648 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:41:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:41:46,652 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:41:46,654 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:41:47,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:41:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:41:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:41:47,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:41:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82be503d]) (logid:ff6484ba) No inactive management server node found 2025-06-29 03:41:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82be503d]) (logid:ff6484ba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:48,035 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-4:[ctx-8d255918]) (logid:b4d6923e) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:41:48,066 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3dd344d8]) (logid:652629f6) ClusterDRS.poll is being called at 2025-06-29T01:42:00+0000 2025-06-29 03:41:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3dd344d8]) (logid:652629f6) Removed 0 old drs migration plans 2025-06-29 03:41:48,139 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-87d3b094]) (logid:d2c51fc3) HA health check task is running... 2025-06-29 03:41:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e2575c33]) (logid:5e359fda) VM scheduler.poll is being called at 2025-06-29T01:42:00+0000 2025-06-29 03:41:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e2575c33]) (logid:5e359fda) Got 0 scheduled jobs to be executed at 2025-06-29T01:42:00+0000 2025-06-29 03:41:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e2575c33]) (logid:5e359fda) Cleaned up 0 VM scheduled job entries 2025-06-29 03:41:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-5c55410c]) (logid:3102eb48) VM Operation Thread Running 2025-06-29 03:41:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad1d46ff]) (logid:6ac79bb6) No inactive management server node found 2025-06-29 03:41:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad1d46ff]) (logid:6ac79bb6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:50,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cd08dd8]) (logid:059641de) No inactive management server node found 2025-06-29 03:41:50,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cd08dd8]) (logid:059641de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:52,141 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-aca409f4]) (logid:a4dc0b5f) HA health check task is running... 2025-06-29 03:41:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0a323c3]) (logid:17f6771e) No inactive management server node found 2025-06-29 03:41:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0a323c3]) (logid:17f6771e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77e67b5]) (logid:2ff49635) No inactive management server node found 2025-06-29 03:41:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77e67b5]) (logid:2ff49635) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:55,222 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8281: Seq 7-8281: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:41:55,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8281 2025-06-29 03:41:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6caed2f2]) (logid:186b76b0) No inactive management server node found 2025-06-29 03:41:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6caed2f2]) (logid:186b76b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:56,142 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a9979317]) (logid:5efccb9e) HA health check task is running... 2025-06-29 03:41:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6988d5b5]) (logid:edbcc7ee) No inactive management server node found 2025-06-29 03:41:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6988d5b5]) (logid:edbcc7ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:58,058 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-e2a8761d]) (logid:ab1b99e1) Starting to remove all volume_stats rows older than [Sat Jun 28 15:41:58 CEST 2025]. 2025-06-29 03:41:58,059 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-e2a8761d]) (logid:ab1b99e1) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:41:58 CEST 2025]. 2025-06-29 03:41:58,063 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-89fb83fc]) (logid:5a0856d8) AutoScaling Monitor is running... 2025-06-29 03:41:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c2e4f20e]) (logid:39fa39e2) Enabled non-edge zones available for scan: 1 2025-06-29 03:41:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c2e4f20e]) (logid:39fa39e2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:41:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c2e4f20e]) (logid:39fa39e2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:41:58,191 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-2a55af97]) (logid:f92ea3fc) Starting to remove all vm_stats rows older than [Sat Jun 28 15:41:58 CEST 2025]. 2025-06-29 03:41:58,215 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-2a55af97]) (logid:f92ea3fc) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:41:58 CEST 2025]. 2025-06-29 03:41:58,244 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:41:58,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:41:58,248 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) Seq 2-4725683383995203618: Routed from 90520738888109 2025-06-29 03:41:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-29ee96fa]) (logid:13d0adca) Enabled non-edge zones available for scan: 1 2025-06-29 03:41:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-29ee96fa]) (logid:13d0adca) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:41:58,293 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) Seq 1-4725683383995203618: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:41:58,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:41:58,296 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) Seq 2-3926857400090361887: Routed from 90520738888109 2025-06-29 03:41:58,341 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7922f902]) (logid:d6826c96) Seq 2-3926857400090361887: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:41:58,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97162839]) (logid:e3010b31) No inactive management server node found 2025-06-29 03:41:58,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97162839]) (logid:e3010b31) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:59,585 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:41:59,589 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:41:59,590 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) Seq 2-4725683383995203619: Routed from 90520738888109 2025-06-29 03:41:59,644 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) Seq 1-4725683383995203619: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:41:59,692 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:41:59,693 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) Seq 2-3926857400090361888: Routed from 90520738888109 2025-06-29 03:41:59,757 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) StorageCollector is running... 2025-06-29 03:41:59,765 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:41:59,765 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:41:59,765 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:41:59,766 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3938116399158788109: Routed from 90520738888109 2025-06-29 03:41:59,787 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bd50b912]) (logid:6eda7b90) Seq 2-3926857400090361888: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:41:59,823 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 4-3938116399158788109: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:41:59,828 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:41:59,828 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:41:59,828 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:41:59,830 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3926857400090361889: Routed from 90520738888109 2025-06-29 03:41:59,843 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3926857400090361889: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:41:59,855 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:41:59,855 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:41:59,855 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:41:59,856 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3926857400090361890: Routed from 90520738888109 2025-06-29 03:41:59,863 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd4f66e8]) (logid:6091331e) No inactive management server node found 2025-06-29 03:41:59,863 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd4f66e8]) (logid:6091331e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:41:59,950 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3926857400090361890: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:41:59,961 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:41:59,961 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:41:59,961 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:41:59,962 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-4725683383995203620: Routed from 90520738888109 2025-06-29 03:42:00,011 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 1-4725683383995203620: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:42:00,013 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:42:00,014 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:42:00,014 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:42:00,015 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3926857400090361891: Routed from 90520738888109 2025-06-29 03:42:00,060 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d6b46a94]) (logid:d227836d) Seq 2-3926857400090361891: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:42:00,144 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-82141289]) (logid:ca2a91de) HA health check task is running... 2025-06-29 03:42:00,221 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8282: Seq 7-8282: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:00,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8282 2025-06-29 03:42:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09ae90b4]) (logid:c0e75d50) No inactive management server node found 2025-06-29 03:42:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09ae90b4]) (logid:c0e75d50) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa03b5fc]) (logid:57eecbce) No inactive management server node found 2025-06-29 03:42:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa03b5fc]) (logid:57eecbce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:04,145 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f2fa91f6]) (logid:e6e17913) HA health check task is running... 2025-06-29 03:42:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf859e6d]) (logid:85549436) No inactive management server node found 2025-06-29 03:42:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf859e6d]) (logid:85549436) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f28cdac]) (logid:ff9f1e30) No inactive management server node found 2025-06-29 03:42:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f28cdac]) (logid:ff9f1e30) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed8af990]) (logid:f39f9641) No inactive management server node found 2025-06-29 03:42:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed8af990]) (logid:f39f9641) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:08,147 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3bf27ee5]) (logid:0c0bff72) HA health check task is running... 2025-06-29 03:42:08,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-865c7ee2]) (logid:8139ebbb) No inactive management server node found 2025-06-29 03:42:08,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-865c7ee2]) (logid:8139ebbb) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:10,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8283: Seq 7-8283: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:10,232 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8283 2025-06-29 03:42:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5855dec5]) (logid:4eb9e173) No inactive management server node found 2025-06-29 03:42:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5855dec5]) (logid:4eb9e173) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcd7889e]) (logid:ee62346e) No inactive management server node found 2025-06-29 03:42:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bcd7889e]) (logid:ee62346e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:12,148 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cb720e1e]) (logid:a989d0bc) HA health check task is running... 2025-06-29 03:42:13,001 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea6067d0]) (logid:5f6d3f21) Checking if any host reservation can be released ... 2025-06-29 03:42:13,007 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea6067d0]) (logid:5f6d3f21) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea6067d0]) (logid:5f6d3f21) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea6067d0]) (logid:5f6d3f21) Done running HostReservationReleaseChecker ... 2025-06-29 03:42:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5828ad0a]) (logid:ff429217) Found 0 routers to update status. 2025-06-29 03:42:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5828ad0a]) (logid:ff429217) Found 0 VPC's to update Redundant State. 2025-06-29 03:42:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5828ad0a]) (logid:ff429217) Found 0 networks to update RvR status. 2025-06-29 03:42:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Running Capacity Checker ... 2025-06-29 03:42:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) recalculating system capacity 2025-06-29 03:42:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Executing cpu/ram capacity update 2025-06-29 03:42:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:13,182 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:13,189 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:42:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:42:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:42:13,201 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:13,211 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:42:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:42:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-14-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:42:13,222 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Done executing cpu/ram capacity update 2025-06-29 03:42:13,222 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Executing storage capacity update 2025-06-29 03:42:13,232 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:42:13,233 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:42:13,233 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:42:13,238 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:42:13,246 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:42:13,246 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:42:13,246 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:42:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:42:13,259 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:42:13,259 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:42:13,259 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:42:13,265 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:42:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:42:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:42:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:42:13,276 DEBUG [c.c.s.StorageManagerImpl] (pool-15-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:42:13,277 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Done executing storage capacity update 2025-06-29 03:42:13,277 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Executing capacity updates for public ip and Vlans 2025-06-29 03:42:13,304 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Done capacity updates for public ip and Vlans 2025-06-29 03:42:13,304 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Executing capacity updates for private ip 2025-06-29 03:42:13,310 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Done executing capacity updates for private ip 2025-06-29 03:42:13,310 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Done recalculating system capacity 2025-06-29 03:42:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-447a769d]) (logid:af055702) No inactive management server node found 2025-06-29 03:42:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-447a769d]) (logid:af055702) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:13,360 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-a40ab6ca]) (logid:9ba278c2) Done running Capacity Checker ... 2025-06-29 03:42:14,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26282e1c]) (logid:b0de3a41) No inactive management server node found 2025-06-29 03:42:14,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26282e1c]) (logid:b0de3a41) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:16,150 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-58470074]) (logid:7c9bc552) HA health check task is running... 2025-06-29 03:42:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3825e23]) (logid:6a921ca3) No inactive management server node found 2025-06-29 03:42:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3825e23]) (logid:6a921ca3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6072f25d]) (logid:b3ed53f6) No inactive management server node found 2025-06-29 03:42:17,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6072f25d]) (logid:b3ed53f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c482402]) (logid:19d863ec) No inactive management server node found 2025-06-29 03:42:19,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c482402]) (logid:19d863ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:20,151 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7fa0f72d]) (logid:7b3f92cb) HA health check task is running... 2025-06-29 03:42:20,223 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8284: Seq 7-8284: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:20,233 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8284 2025-06-29 03:42:20,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8352eb12]) (logid:73ac38aa) No inactive management server node found 2025-06-29 03:42:20,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8352eb12]) (logid:73ac38aa) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:22,365 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d70fb650]) (logid:bdc7a284) No inactive management server node found 2025-06-29 03:42:22,365 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d70fb650]) (logid:bdc7a284) Peer scan is finished. profiler: Done. Duration: 6ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 4ms 2025-06-29 03:42:23,866 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d49106af]) (logid:f4515a6d) No inactive management server node found 2025-06-29 03:42:23,867 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d49106af]) (logid:f4515a6d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:24,153 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c879f9ae]) (logid:02843259) HA health check task is running... 2025-06-29 03:42:25,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbf4869b]) (logid:b9c34fdf) No inactive management server node found 2025-06-29 03:42:25,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbf4869b]) (logid:b9c34fdf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e50ed3e]) (logid:cdf72241) No inactive management server node found 2025-06-29 03:42:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e50ed3e]) (logid:cdf72241) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb15930a]) (logid:0e0b962d) Enabled non-edge zones available for scan: 1 2025-06-29 03:42:28,097 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-eb15930a]) (logid:0e0b962d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:42:28,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-eb15930a]) (logid:0e0b962d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:42:28,154 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-49e1bf48]) (logid:a244955b) HA health check task is running... 2025-06-29 03:42:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-66d0a399]) (logid:d944c3d6) Enabled non-edge zones available for scan: 1 2025-06-29 03:42:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-66d0a399]) (logid:d944c3d6) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:42:28,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d6e1afd]) (logid:7064065c) No inactive management server node found 2025-06-29 03:42:28,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d6e1afd]) (logid:7064065c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:29,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dbde8f9]) (logid:6af071f6) No inactive management server node found 2025-06-29 03:42:29,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dbde8f9]) (logid:6af071f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:30,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8285: Seq 7-8285: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:30,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8285 2025-06-29 03:42:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9742033b]) (logid:5f9bd7f4) No inactive management server node found 2025-06-29 03:42:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9742033b]) (logid:5f9bd7f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:32,155 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3e5dd039]) (logid:858b29e8) HA health check task is running... 2025-06-29 03:42:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d85547a]) (logid:d9eefbb1) No inactive management server node found 2025-06-29 03:42:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d85547a]) (logid:d9eefbb1) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23063a85]) (logid:288e758f) No inactive management server node found 2025-06-29 03:42:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23063a85]) (logid:288e758f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c6b25d1]) (logid:af8de690) No inactive management server node found 2025-06-29 03:42:35,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c6b25d1]) (logid:af8de690) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:36,157 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-bc30d95c]) (logid:87c49eb2) HA health check task is running... 2025-06-29 03:42:37,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-500613b8]) (logid:db900e04) No inactive management server node found 2025-06-29 03:42:37,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-500613b8]) (logid:db900e04) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c59e261d]) (logid:678f80f7) No inactive management server node found 2025-06-29 03:42:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c59e261d]) (logid:678f80f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:40,158 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-777d5a94]) (logid:6fe89b53) HA health check task is running... 2025-06-29 03:42:40,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8286: Seq 7-8286: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:40,233 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8286 2025-06-29 03:42:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc67ed57]) (logid:7bf639ce) No inactive management server node found 2025-06-29 03:42:40,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc67ed57]) (logid:7bf639ce) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ee10145]) (logid:f5f66d96) No inactive management server node found 2025-06-29 03:42:41,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ee10145]) (logid:f5f66d96) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-0e60d8bf]) (logid:c1a09c47) DbCollector is running... 2025-06-29 03:42:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ee0a9d7b]) (logid:e674dc6b) Found 0 routers to update status. 2025-06-29 03:42:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ee0a9d7b]) (logid:e674dc6b) Found 0 VPC's to update Redundant State. 2025-06-29 03:42:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ee0a9d7b]) (logid:e674dc6b) Found 0 networks to update RvR status. 2025-06-29 03:42:43,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c337800]) (logid:9ca29eef) No inactive management server node found 2025-06-29 03:42:43,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c337800]) (logid:9ca29eef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) ManagementServerCollector is running... 2025-06-29 03:42:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) DB properties were already loaded 2025-06-29 03:42:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:42:44,059 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Execution of process [4065050] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:42:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Exit value of process [4065050] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:42:44,059 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Process [4065050] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:42:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) usage install: null 2025-06-29 03:42:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Sessions active: 0 2025-06-29 03:42:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:42:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065053] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:42:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:42:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065058] for command [/bin/bash -c uname -r ]. 2025-06-29 03:42:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:42:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065059] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:42:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) system memory from /proc: 4887515136 2025-06-29 03:42:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:42:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065064] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:42:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) free memory from /proc: 149852160 2025-06-29 03:42:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:42:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065069] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:42:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) used memory from /proc: 1072224 2025-06-29 03:42:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:42:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065070] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:42:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:42:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065071] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:42:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:42:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065072] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:42:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:42:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065079] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:42:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:42:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Successfully executed process [4065080] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail buffersdirect.capacity, 209730 2025-06-29 03:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail buffersdirect.count, 68 2025-06-29 03:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail buffersdirect.used, 209730 2025-06-29 03:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail buffersmapped.capacity, 0 2025-06-29 03:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail buffersmapped.count, 0 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail buffersmapped.used, 0 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail jvmuptime, 423835 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memoryheap.committed, 517996544 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memoryheap.init, 77594624 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memoryheap.usage, 0.24206820820360778 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorynon-heap.committed, 186777600 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorynon-heap.max, -1 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorynon-heap.usage, -1.84542016E8 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorynon-heap.used, 184542016 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3031798245614035 2025-06-29 03:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.08814941015729139 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.1914834455293765 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.Compressed-Class-Space.usage, 0.01215384155511856 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.Metaspace.usage, 0.994375420810514 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.PS-Eden-Space.usage, 0.4251612865052104 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorytotal.committed, 704774144 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorytotal.init, 85262336 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorytotal.max, 1908932607 2025-06-29 03:42:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail memorytotal.used, 646633912 2025-06-29 03:42:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail threadsdeadlocks, [] 2025-06-29 03:42:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail threadsnew.count, 0 2025-06-29 03:42:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:42:44,138 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-32358ec4]) (logid:5b1d8444) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":56690464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":423762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":645820136,"totalCommitted":704774144,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":462091896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":149852160,"systemMemoryUsed":1072224,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.22,0.21],"systemCyclesUsage":[66396754,53821577,4588781858],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:42:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 13, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":56690464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":423762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":645820136,"totalCommitted":704774144,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":462091896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":149852160,"systemMemoryUsed":1072224,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.22,0.21],"systemCyclesUsage":[66396754,53821577,4588781858],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:42:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:42:44,145 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-8:[ctx-41f47320]) (logid:2a42368e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":56690464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":423762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":645820136,"totalCommitted":704774144,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":462091896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":149852160,"systemMemoryUsed":1072224,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.22,0.21],"systemCyclesUsage":[66396754,53821577,4588781858],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:42:44,146 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 03:42:44,146 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 13, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":56690464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":423762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":645820136,"totalCommitted":704774144,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":462091896,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":149852160,"systemMemoryUsed":1072224,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.22,0.21],"systemCyclesUsage":[66396754,53821577,4588781858],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:42:44,160 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-746db4a8]) (logid:bd3ce14c) HA health check task is running... 2025-06-29 03:42:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5334dcd7]) (logid:b5cfb999) No inactive management server node found 2025-06-29 03:42:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5334dcd7]) (logid:b5cfb999) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:45,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8287: Seq 7-8287: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:45,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8287 2025-06-29 03:42:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:42:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:42:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f7beaf0]) (logid:aa308b47) No inactive management server node found 2025-06-29 03:42:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f7beaf0]) (logid:aa308b47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:42:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:42:47,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:42:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:42:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f95e366]) (logid:c34af5a7) No inactive management server node found 2025-06-29 03:42:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f95e366]) (logid:c34af5a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-06a6f70f]) (logid:041a2fbf) ClusterDRS.poll is being called at 2025-06-29T01:43:00+0000 2025-06-29 03:42:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-06a6f70f]) (logid:041a2fbf) Removed 0 old drs migration plans 2025-06-29 03:42:48,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-98a33177]) (logid:bfb5d7a6) Resetting hosts suitable for reconnect 2025-06-29 03:42:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-98a33177]) (logid:bfb5d7a6) Completed resetting hosts suitable for reconnect 2025-06-29 03:42:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-98a33177]) (logid:bfb5d7a6) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:42:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-98a33177]) (logid:bfb5d7a6) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:42:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-98a33177]) (logid:bfb5d7a6) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:42:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-98a33177]) (logid:bfb5d7a6) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:42:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-114f02a3]) (logid:deca0655) VM scheduler.poll is being called at 2025-06-29T01:43:00+0000 2025-06-29 03:42:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-114f02a3]) (logid:deca0655) Got 0 scheduled jobs to be executed at 2025-06-29T01:43:00+0000 2025-06-29 03:42:48,161 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3bd1dbea]) (logid:5b117122) HA health check task is running... 2025-06-29 03:42:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-114f02a3]) (logid:deca0655) Cleaned up 0 VM scheduled job entries 2025-06-29 03:42:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-879d5d7e]) (logid:e406a568) VM Operation Thread Running 2025-06-29 03:42:49,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a38d4b27]) (logid:89e3bed8) No inactive management server node found 2025-06-29 03:42:49,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a38d4b27]) (logid:89e3bed8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16a25c33]) (logid:464fe31b) No inactive management server node found 2025-06-29 03:42:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16a25c33]) (logid:464fe31b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:52,162 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6df2bc35]) (logid:946a53df) HA health check task is running... 2025-06-29 03:42:52,368 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53caa6e9]) (logid:c8b72249) No inactive management server node found 2025-06-29 03:42:52,368 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53caa6e9]) (logid:c8b72249) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0dae73c]) (logid:68ad576a) No inactive management server node found 2025-06-29 03:42:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0dae73c]) (logid:68ad576a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:42:55,225 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8289: Seq 7-8289: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:42:55,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8289 2025-06-29 03:42:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-545afab8]) (logid:caa66362) No inactive management server node found 2025-06-29 03:42:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-545afab8]) (logid:caa66362) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:56,163 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d4e6afd2]) (logid:0cda50f3) HA health check task is running... 2025-06-29 03:42:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1325cde1]) (logid:e85be02e) No inactive management server node found 2025-06-29 03:42:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1325cde1]) (logid:e85be02e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:58,061 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-04ffaeba]) (logid:6a2bd239) Starting to remove all volume_stats rows older than [Sat Jun 28 15:42:58 CEST 2025]. 2025-06-29 03:42:58,062 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-04ffaeba]) (logid:6a2bd239) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:42:58 CEST 2025]. 2025-06-29 03:42:58,065 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-60ffd556]) (logid:bebd6669) AutoScaling Monitor is running... 2025-06-29 03:42:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a96d06fe]) (logid:37256239) Enabled non-edge zones available for scan: 1 2025-06-29 03:42:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a96d06fe]) (logid:37256239) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:42:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a96d06fe]) (logid:37256239) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:42:58,217 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-85ebc570]) (logid:93a62b8a) Starting to remove all vm_stats rows older than [Sat Jun 28 15:42:58 CEST 2025]. 2025-06-29 03:42:58,242 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-85ebc570]) (logid:93a62b8a) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:42:58 CEST 2025]. 2025-06-29 03:42:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4be23485]) (logid:48d1a411) Enabled non-edge zones available for scan: 1 2025-06-29 03:42:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4be23485]) (logid:48d1a411) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:42:58,345 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:42:58,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:42:58,349 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) Seq 2-4725683383995203621: Routed from 90520738888109 2025-06-29 03:42:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aee0fe2b]) (logid:93a037f2) No inactive management server node found 2025-06-29 03:42:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aee0fe2b]) (logid:93a037f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:58,393 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) Seq 1-4725683383995203621: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:42:58,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:42:58,395 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) Seq 2-3926857400090361892: Routed from 90520738888109 2025-06-29 03:42:58,440 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-929d0aec]) (logid:4f5b5b38) Seq 2-3926857400090361892: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:42:59,842 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:42:59,846 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:42:59,847 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) Seq 2-4725683383995203622: Routed from 90520738888109 2025-06-29 03:42:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a509bfb]) (logid:372f369a) No inactive management server node found 2025-06-29 03:42:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a509bfb]) (logid:372f369a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:42:59,901 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) Seq 1-4725683383995203622: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:42:59,936 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:42:59,937 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) Seq 2-3926857400090361893: Routed from 90520738888109 2025-06-29 03:43:00,049 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-09957a59]) (logid:e51e5738) Seq 2-3926857400090361893: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:43:00,060 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) StorageCollector is running... 2025-06-29 03:43:00,069 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:43:00,069 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:43:00,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:43:00,070 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-3938116399158788110: Routed from 90520738888109 2025-06-29 03:43:00,132 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 4-3938116399158788110: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:43:00,137 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:43:00,137 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:43:00,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:43:00,139 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-4725683383995203623: Routed from 90520738888109 2025-06-29 03:43:00,165 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ba1bb290]) (logid:b47822be) HA health check task is running... 2025-06-29 03:43:00,188 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 1-4725683383995203623: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:43:00,199 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:43:00,199 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:43:00,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:43:00,201 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-3926857400090361894: Routed from 90520738888109 2025-06-29 03:43:00,215 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-3926857400090361894: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:43:00,226 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:43:00,226 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:43:00,226 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:43:00,228 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-4725683383995203624: Routed from 90520738888109 2025-06-29 03:43:00,272 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 1-4725683383995203624: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:43:00,274 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:43:00,274 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:43:00,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:43:00,276 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-3926857400090361895: Routed from 90520738888109 2025-06-29 03:43:00,321 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-267392f7]) (logid:a17cc96b) Seq 2-3926857400090361895: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:43:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a477f644]) (logid:88ef04a5) No inactive management server node found 2025-06-29 03:43:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a477f644]) (logid:88ef04a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51fb50b8]) (logid:65701544) No inactive management server node found 2025-06-29 03:43:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51fb50b8]) (logid:65701544) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:04,167 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8c2f51fe]) (logid:e72290ff) HA health check task is running... 2025-06-29 03:43:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8018ba30]) (logid:3c1a467e) No inactive management server node found 2025-06-29 03:43:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8018ba30]) (logid:3c1a467e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:05,225 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8290: Seq 7-8290: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:43:05,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8290 2025-06-29 03:43:05,875 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8c9cade]) (logid:c35e1f3f) No inactive management server node found 2025-06-29 03:43:05,875 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8c9cade]) (logid:c35e1f3f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:07,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c12cb4c0]) (logid:a9cdc371) No inactive management server node found 2025-06-29 03:43:07,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c12cb4c0]) (logid:a9cdc371) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:08,168 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d647e93d]) (logid:b002b828) HA health check task is running... 2025-06-29 03:43:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffc940ce]) (logid:3b2c69f6) No inactive management server node found 2025-06-29 03:43:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffc940ce]) (logid:3b2c69f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d27dc16]) (logid:71a54941) No inactive management server node found 2025-06-29 03:43:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d27dc16]) (logid:71a54941) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:43:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d84faa8]) (logid:42dd3480) No inactive management server node found 2025-06-29 03:43:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d84faa8]) (logid:42dd3480) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:12,169 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0c3c9934]) (logid:6a669873) HA health check task is running... 2025-06-29 03:43:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d88ce95e]) (logid:469e0571) Found 0 routers to update status. 2025-06-29 03:43:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d88ce95e]) (logid:469e0571) Found 0 VPC's to update Redundant State. 2025-06-29 03:43:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d88ce95e]) (logid:469e0571) Found 0 networks to update RvR status. 2025-06-29 03:43:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb8b8b05]) (logid:514493d8) No inactive management server node found 2025-06-29 03:43:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb8b8b05]) (logid:514493d8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f418a656]) (logid:78b24e38) No inactive management server node found 2025-06-29 03:43:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f418a656]) (logid:78b24e38) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:15,226 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8291: Seq 7-8291: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:43:15,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8291 2025-06-29 03:43:16,171 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ca48067e]) (logid:dfdda73e) HA health check task is running... 2025-06-29 03:43:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ab75f41]) (logid:f1a506b8) No inactive management server node found 2025-06-29 03:43:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ab75f41]) (logid:f1a506b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c82bf15]) (logid:452fddfd) No inactive management server node found 2025-06-29 03:43:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c82bf15]) (logid:452fddfd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9408976a]) (logid:55fdec0f) No inactive management server node found 2025-06-29 03:43:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9408976a]) (logid:55fdec0f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:20,173 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-34db2622]) (logid:fba47bc1) HA health check task is running... 2025-06-29 03:43:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-955e62ff]) (logid:1878eb67) No inactive management server node found 2025-06-29 03:43:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-955e62ff]) (logid:1878eb67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d562533e]) (logid:33d2fb78) No inactive management server node found 2025-06-29 03:43:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d562533e]) (logid:33d2fb78) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56b1a629]) (logid:1dbce71b) No inactive management server node found 2025-06-29 03:43:23,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56b1a629]) (logid:1dbce71b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:24,174 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ce551328]) (logid:6cd21824) HA health check task is running... 2025-06-29 03:43:25,226 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8292: Seq 7-8292: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:43:25,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8292 2025-06-29 03:43:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84c11f4f]) (logid:c4585e67) No inactive management server node found 2025-06-29 03:43:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84c11f4f]) (logid:c4585e67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:26,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60e76336]) (logid:3cb8f64c) No inactive management server node found 2025-06-29 03:43:26,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60e76336]) (logid:3cb8f64c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9dabf857]) (logid:b0fb298a) Enabled non-edge zones available for scan: 1 2025-06-29 03:43:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9dabf857]) (logid:b0fb298a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:43:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9dabf857]) (logid:b0fb298a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:43:28,176 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-fbd298fc]) (logid:5c8571ff) HA health check task is running... 2025-06-29 03:43:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-efc5ea2f]) (logid:84afd730) Enabled non-edge zones available for scan: 1 2025-06-29 03:43:28,284 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-efc5ea2f]) (logid:84afd730) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:43:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16be1a91]) (logid:fcca750c) No inactive management server node found 2025-06-29 03:43:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16be1a91]) (logid:fcca750c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:29,875 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3d62c13]) (logid:89fc7498) No inactive management server node found 2025-06-29 03:43:29,876 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3d62c13]) (logid:89fc7498) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:31,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bb3f935]) (logid:baed5fa7) No inactive management server node found 2025-06-29 03:43:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bb3f935]) (logid:baed5fa7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:32,177 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f08c85a9]) (logid:fc9c2bc2) HA health check task is running... 2025-06-29 03:43:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05a9ee53]) (logid:3103c8e7) No inactive management server node found 2025-06-29 03:43:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05a9ee53]) (logid:3103c8e7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d36af1b]) (logid:d6087c8c) No inactive management server node found 2025-06-29 03:43:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d36af1b]) (logid:d6087c8c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:35,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8293: Seq 7-8293: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:43:35,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8293 2025-06-29 03:43:35,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfc50e91]) (logid:5a45f52f) No inactive management server node found 2025-06-29 03:43:35,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfc50e91]) (logid:5a45f52f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:36,179 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f83d8b39]) (logid:d026b20d) HA health check task is running... 2025-06-29 03:43:37,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cc670db]) (logid:0d41dea6) No inactive management server node found 2025-06-29 03:43:37,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cc670db]) (logid:0d41dea6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:43:38,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a21d5fa2]) (logid:d8e39229) No inactive management server node found 2025-06-29 03:43:38,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a21d5fa2]) (logid:d8e39229) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:40,180 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b1666604]) (logid:a5a3e945) HA health check task is running... 2025-06-29 03:43:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9785a4c7]) (logid:1f8325ab) No inactive management server node found 2025-06-29 03:43:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9785a4c7]) (logid:1f8325ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:41,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-294a7e5c]) (logid:3d6eac3a) No inactive management server node found 2025-06-29 03:43:41,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-294a7e5c]) (logid:3d6eac3a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-16229756]) (logid:07c0af66) DbCollector is running... 2025-06-29 03:43:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c2b2c868]) (logid:0ea008bd) Found 0 routers to update status. 2025-06-29 03:43:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c2b2c868]) (logid:0ea008bd) Found 0 VPC's to update Redundant State. 2025-06-29 03:43:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c2b2c868]) (logid:0ea008bd) Found 0 networks to update RvR status. 2025-06-29 03:43:43,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-740a1c00]) (logid:caca5adb) No inactive management server node found 2025-06-29 03:43:43,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-740a1c00]) (logid:caca5adb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) ManagementServerCollector is running... 2025-06-29 03:43:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) DB properties were already loaded 2025-06-29 03:43:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:43:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Execution of process [4065084] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:43:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Exit value of process [4065084] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:43:44,058 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Process [4065084] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:43:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) usage install: null 2025-06-29 03:43:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Sessions active: 0 2025-06-29 03:43:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:43:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065087] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:43:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:43:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065092] for command [/bin/bash -c uname -r ]. 2025-06-29 03:43:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:43:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065093] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:43:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) system memory from /proc: 4887515136 2025-06-29 03:43:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:43:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065098] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:43:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) free memory from /proc: 147558400 2025-06-29 03:43:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:43:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065103] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:43:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) used memory from /proc: 1072352 2025-06-29 03:43:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:43:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065104] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:43:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:43:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065105] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:43:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:43:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065106] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:43:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:43:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065113] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:43:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:43:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Successfully executed process [4065114] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:43:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail buffersdirect.capacity, 216160 2025-06-29 03:43:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail buffersdirect.count, 69 2025-06-29 03:43:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail buffersdirect.used, 216160 2025-06-29 03:43:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail buffersmapped.capacity, 0 2025-06-29 03:43:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail buffersmapped.count, 0 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail buffersmapped.used, 0 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail gcPS-Scavenge.count, 104 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail gcPS-Scavenge.time, 463 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail jvmuptime, 483835 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memoryheap.committed, 517996544 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memoryheap.init, 77594624 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memoryheap.usage, 0.25368205350495016 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorynon-heap.committed, 187105280 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorynon-heap.max, -1 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorynon-heap.usage, -1.8506236E8 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorynon-heap.used, 185062360 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3031798245614035 2025-06-29 03:43:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.08874300186616903 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19429633085613357 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.Compressed-Class-Space.usage, 0.012157909572124481 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.Metaspace.usage, 0.9941601186718374 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.PS-Eden-Space.usage, 0.45918060327285926 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.PS-Old-Gen.usage, 0.11201974875060074 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorypools.PS-Survivor-Space.usage, 0.999339002244016 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorytotal.committed, 705101824 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorytotal.init, 85262336 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorytotal.max, 1908932607 2025-06-29 03:43:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail memorytotal.used, 669328784 2025-06-29 03:43:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail threadsdeadlocks, [] 2025-06-29 03:43:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail threadsnew.count, 0 2025-06-29 03:43:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:43:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-d5377021]) (logid:2ba46b09) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":34520464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":483761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":668521024,"totalCommitted":705036288,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":484261944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":147558400,"systemMemoryUsed":1072352,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.24,0.21],"systemCyclesUsage":[66397089,53821873,4588804980],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:43:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 15, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":34520464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":483761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":668521024,"totalCommitted":705036288,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":484261944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":147558400,"systemMemoryUsed":1072352,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.24,0.21],"systemCyclesUsage":[66397089,53821873,4588804980],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:43:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:43:44,146 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-8:[ctx-d9c306ea]) (logid:f6139f58) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":34520464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":483761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":668521024,"totalCommitted":705036288,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":484261944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":147558400,"systemMemoryUsed":1072352,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.24,0.21],"systemCyclesUsage":[66397089,53821873,4588804980],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:43:44,146 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:43:44,147 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 15, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":517996544,"freeJvmMemoryBytes":34520464,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":483761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.22,"totalInit":85262336,"totalUsed":668521024,"totalCommitted":705036288,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":484261944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":147558400,"systemMemoryUsed":1072352,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.22,0.24,0.21],"systemCyclesUsage":[66397089,53821873,4588804980],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:43:44,182 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-13f8c6a4]) (logid:1d23f4a3) HA health check task is running... 2025-06-29 03:43:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-229b31bc]) (logid:9c8ac9ac) No inactive management server node found 2025-06-29 03:43:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-229b31bc]) (logid:9c8ac9ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:45,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8294: Seq 7-8294: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:43:45,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8294 2025-06-29 03:43:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:43:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:43:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca70c2a7]) (logid:dc5c6a96) No inactive management server node found 2025-06-29 03:43:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca70c2a7]) (logid:dc5c6a96) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:43:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:43:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:43:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:43:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:43:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:43:47,074 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:43:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:43:47,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81639a8c]) (logid:df71e388) No inactive management server node found 2025-06-29 03:43:47,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81639a8c]) (logid:df71e388) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9b8c58e6]) (logid:87cb5d38) ClusterDRS.poll is being called at 2025-06-29T01:44:00+0000 2025-06-29 03:43:48,073 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-9b8c58e6]) (logid:87cb5d38) Removed 0 old drs migration plans 2025-06-29 03:43:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a7befd4f]) (logid:e8b68a1c) VM scheduler.poll is being called at 2025-06-29T01:44:00+0000 2025-06-29 03:43:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a7befd4f]) (logid:e8b68a1c) Got 0 scheduled jobs to be executed at 2025-06-29T01:44:00+0000 2025-06-29 03:43:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a7befd4f]) (logid:e8b68a1c) Cleaned up 0 VM scheduled job entries 2025-06-29 03:43:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-c10fc55a]) (logid:d12998ae) VM Operation Thread Running 2025-06-29 03:43:48,183 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-fc1ca822]) (logid:bc5f384f) HA health check task is running... 2025-06-29 03:43:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc169cc7]) (logid:e3334522) No inactive management server node found 2025-06-29 03:43:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc169cc7]) (logid:e3334522) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:50,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43e64a5c]) (logid:ce9ca1ff) No inactive management server node found 2025-06-29 03:43:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43e64a5c]) (logid:ce9ca1ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:52,185 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-28e0541f]) (logid:b7d9620d) HA health check task is running... 2025-06-29 03:43:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fc4ce6f]) (logid:66c1a647) No inactive management server node found 2025-06-29 03:43:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fc4ce6f]) (logid:66c1a647) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:53,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee604402]) (logid:1a9be8c1) No inactive management server node found 2025-06-29 03:43:53,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee604402]) (logid:1a9be8c1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:55,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8296: Seq 7-8296: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:43:55,248 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8296 2025-06-29 03:43:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7139a05]) (logid:50180ca3) No inactive management server node found 2025-06-29 03:43:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7139a05]) (logid:50180ca3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:56,186 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-31e9465c]) (logid:ef635d0a) HA health check task is running... 2025-06-29 03:43:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fedfeef]) (logid:107e4ed4) No inactive management server node found 2025-06-29 03:43:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fedfeef]) (logid:107e4ed4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:58,064 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-0e3bf6ff]) (logid:3f792b3d) Starting to remove all volume_stats rows older than [Sat Jun 28 15:43:58 CEST 2025]. 2025-06-29 03:43:58,065 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-0e3bf6ff]) (logid:3f792b3d) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:43:58 CEST 2025]. 2025-06-29 03:43:58,067 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-5c1afe7c]) (logid:76f3e293) AutoScaling Monitor is running... 2025-06-29 03:43:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d09a2328]) (logid:7b2da621) Enabled non-edge zones available for scan: 1 2025-06-29 03:43:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d09a2328]) (logid:7b2da621) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:43:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d09a2328]) (logid:7b2da621) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:43:58,244 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-7f1a9bbb]) (logid:759cca99) Starting to remove all vm_stats rows older than [Sat Jun 28 15:43:58 CEST 2025]. 2025-06-29 03:43:58,269 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-7f1a9bbb]) (logid:759cca99) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:43:58 CEST 2025]. 2025-06-29 03:43:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c86e6108]) (logid:457bea86) Enabled non-edge zones available for scan: 1 2025-06-29 03:43:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c86e6108]) (logid:457bea86) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:43:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1e400c6]) (logid:f5c822b9) No inactive management server node found 2025-06-29 03:43:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1e400c6]) (logid:f5c822b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:58,443 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:43:58,445 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:43:58,447 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) Seq 2-4725683383995203625: Routed from 90520738888109 2025-06-29 03:43:58,491 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) Seq 1-4725683383995203625: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:43:58,493 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:43:58,495 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) Seq 2-3926857400090361896: Routed from 90520738888109 2025-06-29 03:43:58,540 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-74b9991d]) (logid:ee1bfcca) Seq 2-3926857400090361896: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:43:59,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e39b8cb8]) (logid:fc42687a) No inactive management server node found 2025-06-29 03:43:59,894 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e39b8cb8]) (logid:fc42687a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:43:59,894 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:43:59,895 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:44:00,105 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:44:00,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:44:00,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) Seq 2-4725683383995203626: Routed from 90520738888109 2025-06-29 03:44:00,165 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) Seq 1-4725683383995203626: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:44:00,188 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3dd1f6ac]) (logid:6f392063) HA health check task is running... 2025-06-29 03:44:00,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:44:00,216 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) Seq 2-3926857400090361897: Routed from 90520738888109 2025-06-29 03:44:00,311 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-86100660]) (logid:903caf86) Seq 2-3926857400090361897: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:44:00,322 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) StorageCollector is running... 2025-06-29 03:44:00,332 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:44:00,332 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:44:00,332 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:44:00,334 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3938116399158788111: Routed from 90520738888109 2025-06-29 03:44:00,407 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 4-3938116399158788111: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:44:00,413 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:44:00,413 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:44:00,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:44:00,415 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3926857400090361898: Routed from 90520738888109 2025-06-29 03:44:00,429 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3926857400090361898: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:44:00,442 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:44:00,442 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:44:00,442 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:44:00,443 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3926857400090361899: Routed from 90520738888109 2025-06-29 03:44:00,537 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3926857400090361899: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:44:00,548 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:44:00,548 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:44:00,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:44:00,549 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-4725683383995203627: Routed from 90520738888109 2025-06-29 03:44:00,594 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 1-4725683383995203627: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:44:00,596 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:44:00,597 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:44:00,597 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:44:00,598 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3926857400090361900: Routed from 90520738888109 2025-06-29 03:44:00,643 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a7fd57a6]) (logid:64b29e41) Seq 2-3926857400090361900: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:44:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c09a6f7]) (logid:943ada22) No inactive management server node found 2025-06-29 03:44:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c09a6f7]) (logid:943ada22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e71e0d1]) (logid:380263ae) No inactive management server node found 2025-06-29 03:44:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e71e0d1]) (logid:380263ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:04,190 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e1c1bb04]) (logid:a86d354e) HA health check task is running... 2025-06-29 03:44:04,368 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0472021d]) (logid:7efd7ae0) No inactive management server node found 2025-06-29 03:44:04,368 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0472021d]) (logid:7efd7ae0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:05,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8297: Seq 7-8297: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:44:05,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8297 2025-06-29 03:44:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26d38730]) (logid:005e2572) No inactive management server node found 2025-06-29 03:44:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26d38730]) (logid:005e2572) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:07,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ab58d32]) (logid:8dd0d2d5) No inactive management server node found 2025-06-29 03:44:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ab58d32]) (logid:8dd0d2d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:08,191 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a99c4e1a]) (logid:ef9a4c45) HA health check task is running... 2025-06-29 03:44:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6faca724]) (logid:e79a636f) No inactive management server node found 2025-06-29 03:44:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6faca724]) (logid:e79a636f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb38cd6]) (logid:dc23e24f) No inactive management server node found 2025-06-29 03:44:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb38cd6]) (logid:dc23e24f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cf2da0f]) (logid:a044382d) No inactive management server node found 2025-06-29 03:44:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cf2da0f]) (logid:a044382d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:44:12,193 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2b791ae3]) (logid:69a7a985) HA health check task is running... 2025-06-29 03:44:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e3cc052b]) (logid:15b633ab) Found 0 routers to update status. 2025-06-29 03:44:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e3cc052b]) (logid:15b633ab) Found 0 VPC's to update Redundant State. 2025-06-29 03:44:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e3cc052b]) (logid:15b633ab) Found 0 networks to update RvR status. 2025-06-29 03:44:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a15fa0f3]) (logid:18bdcd3f) No inactive management server node found 2025-06-29 03:44:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a15fa0f3]) (logid:18bdcd3f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4ab08e7]) (logid:d55f52a5) No inactive management server node found 2025-06-29 03:44:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4ab08e7]) (logid:d55f52a5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:15,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8298: Seq 7-8298: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:44:15,250 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8298 2025-06-29 03:44:16,194 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c3a4c58d]) (logid:4a4becff) HA health check task is running... 2025-06-29 03:44:16,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3662ac41]) (logid:e20419fc) No inactive management server node found 2025-06-29 03:44:16,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3662ac41]) (logid:e20419fc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:17,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97acbbf2]) (logid:e46cbf43) No inactive management server node found 2025-06-29 03:44:17,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97acbbf2]) (logid:e46cbf43) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:18,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9992734a]) (logid:59161c8c) Resetting hosts suitable for reconnect 2025-06-29 03:44:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9992734a]) (logid:59161c8c) Completed resetting hosts suitable for reconnect 2025-06-29 03:44:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9992734a]) (logid:59161c8c) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:44:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9992734a]) (logid:59161c8c) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:44:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9992734a]) (logid:59161c8c) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:44:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9992734a]) (logid:59161c8c) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:44:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f0dc91a]) (logid:e785d694) No inactive management server node found 2025-06-29 03:44:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f0dc91a]) (logid:e785d694) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:20,195 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e964374f]) (logid:248b0508) HA health check task is running... 2025-06-29 03:44:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d54aea97]) (logid:096820b6) No inactive management server node found 2025-06-29 03:44:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d54aea97]) (logid:096820b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-afcbcd50]) (logid:3059280d) No inactive management server node found 2025-06-29 03:44:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-afcbcd50]) (logid:3059280d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e1d2223]) (logid:0237e80c) No inactive management server node found 2025-06-29 03:44:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e1d2223]) (logid:0237e80c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:24,197 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-27dd3d36]) (logid:57ba8bc9) HA health check task is running... 2025-06-29 03:44:25,229 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8299: Seq 7-8299: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:44:25,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8299 2025-06-29 03:44:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85fea904]) (logid:b44671ab) No inactive management server node found 2025-06-29 03:44:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85fea904]) (logid:b44671ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:26,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea6a020f]) (logid:d54425df) No inactive management server node found 2025-06-29 03:44:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea6a020f]) (logid:d54425df) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c4aa4a8]) (logid:361c044e) Enabled non-edge zones available for scan: 1 2025-06-29 03:44:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0c4aa4a8]) (logid:361c044e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:44:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c4aa4a8]) (logid:361c044e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:44:28,198 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-65a9c7ea]) (logid:17e173ff) HA health check task is running... 2025-06-29 03:44:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-054f76b1]) (logid:36f5798f) Enabled non-edge zones available for scan: 1 2025-06-29 03:44:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-054f76b1]) (logid:36f5798f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:44:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faa9e9ad]) (logid:64290faf) No inactive management server node found 2025-06-29 03:44:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faa9e9ad]) (logid:64290faf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b96900c9]) (logid:e1b4b92e) No inactive management server node found 2025-06-29 03:44:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b96900c9]) (logid:e1b4b92e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be8b1ab7]) (logid:6336afb0) No inactive management server node found 2025-06-29 03:44:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be8b1ab7]) (logid:6336afb0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:32,199 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e08e8159]) (logid:7f905a6c) HA health check task is running... 2025-06-29 03:44:32,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6404b104]) (logid:efcf324f) No inactive management server node found 2025-06-29 03:44:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6404b104]) (logid:efcf324f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6fb4c8b]) (logid:43af58cd) No inactive management server node found 2025-06-29 03:44:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6fb4c8b]) (logid:43af58cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:35,229 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8300: Seq 7-8300: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:44:35,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8300 2025-06-29 03:44:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc3f68dc]) (logid:fc5d6a0c) No inactive management server node found 2025-06-29 03:44:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc3f68dc]) (logid:fc5d6a0c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:36,201 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e838b199]) (logid:f9af3e41) HA health check task is running... 2025-06-29 03:44:37,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e20edea1]) (logid:10638b85) No inactive management server node found 2025-06-29 03:44:37,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e20edea1]) (logid:10638b85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0463a239]) (logid:79c536dd) No inactive management server node found 2025-06-29 03:44:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0463a239]) (logid:79c536dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:40,202 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2481c9f9]) (logid:6806d9af) HA health check task is running... 2025-06-29 03:44:40,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fc78ec3]) (logid:e2500158) No inactive management server node found 2025-06-29 03:44:40,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fc78ec3]) (logid:e2500158) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:44:41,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-847b8e90]) (logid:345a4cd7) No inactive management server node found 2025-06-29 03:44:41,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-847b8e90]) (logid:345a4cd7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-2b6eec92]) (logid:196697aa) DbCollector is running... 2025-06-29 03:44:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a15cda38]) (logid:3121b12b) Found 0 routers to update status. 2025-06-29 03:44:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a15cda38]) (logid:3121b12b) Found 0 VPC's to update Redundant State. 2025-06-29 03:44:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a15cda38]) (logid:3121b12b) Found 0 networks to update RvR status. 2025-06-29 03:44:43,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f0749cf]) (logid:92a0642b) No inactive management server node found 2025-06-29 03:44:43,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f0749cf]) (logid:92a0642b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) ManagementServerCollector is running... 2025-06-29 03:44:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) DB properties were already loaded 2025-06-29 03:44:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:44:44,058 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Execution of process [4065148] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:44:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Exit value of process [4065148] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:44:44,058 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Process [4065148] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:44:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) usage install: null 2025-06-29 03:44:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Sessions active: 0 2025-06-29 03:44:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:44:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065151] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:44:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:44:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065156] for command [/bin/bash -c uname -r ]. 2025-06-29 03:44:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:44:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065157] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:44:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) system memory from /proc: 4887515136 2025-06-29 03:44:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:44:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065162] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:44:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) free memory from /proc: 164229120 2025-06-29 03:44:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:44:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065167] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:44:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) used memory from /proc: 1069036 2025-06-29 03:44:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:44:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065168] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:44:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:44:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065169] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:44:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:44:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065170] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:44:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:44:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065177] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:44:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:44:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Successfully executed process [4065178] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail buffersdirect.capacity, 216280 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail buffersdirect.count, 69 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail buffersdirect.used, 216280 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail buffersmapped.capacity, 0 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail buffersmapped.count, 0 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail buffersmapped.used, 0 2025-06-29 03:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail jvmuptime, 543836 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memoryheap.committed, 545259520 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memoryheap.init, 77594624 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memoryheap.usage, 0.12006872900564963 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorynon-heap.committed, 187498496 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorynon-heap.max, -1 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorynon-heap.usage, -1.85518128E8 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorynon-heap.used, 185518128 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3031798245614035 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09029258864302853 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19601467990802146 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012160994112491608 2025-06-29 03:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.Metaspace.usage, 0.9945056616199194 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.PS-Eden-Space.usage, 0.0530788865054037 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorytotal.committed, 732758016 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorytotal.init, 85262336 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorytotal.max, 1908932607 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail memorytotal.used, 414721240 2025-06-29 03:44:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail threadsdeadlocks, [] 2025-06-29 03:44:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail threadsnew.count, 0 2025-06-29 03:44:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:44:44,138 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-76fd25fe]) (logid:7bd3677b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":316812064,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":543762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":413964520,"totalCommitted":732758016,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":229203112,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":164229120,"systemMemoryUsed":1069036,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.19,0.2],"systemCyclesUsage":[66397494,53822181,4588828009],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:44:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 17, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":316812064,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":543762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":413964520,"totalCommitted":732758016,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":229203112,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":164229120,"systemMemoryUsed":1069036,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.19,0.2],"systemCyclesUsage":[66397494,53822181,4588828009],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:44:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:44:44,147 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-8:[ctx-8927ce64]) (logid:214d8eeb) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":316812064,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":543762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":413964520,"totalCommitted":732758016,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":229203112,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":164229120,"systemMemoryUsed":1069036,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.19,0.2],"systemCyclesUsage":[66397494,53822181,4588828009],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:44:44,148 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:44:44,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 17, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":316812064,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":543762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":413964520,"totalCommitted":732758016,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":229203112,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":194,"threadsWaitingCount":102,"systemMemoryTotal":4887515136,"systemMemoryFree":164229120,"systemMemoryUsed":1069036,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.19,0.2],"systemCyclesUsage":[66397494,53822181,4588828009],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:44:44,204 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-11d17289]) (logid:64a9a2fe) HA health check task is running... 2025-06-29 03:44:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa10ef12]) (logid:6f25b2f2) No inactive management server node found 2025-06-29 03:44:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa10ef12]) (logid:6f25b2f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:45,230 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8301: Seq 7-8301: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:44:45,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8301 2025-06-29 03:44:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:44:45,926 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:44:46,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4c5f795]) (logid:d81e1210) No inactive management server node found 2025-06-29 03:44:46,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4c5f795]) (logid:d81e1210) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:44:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:44:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:44:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:44:47,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:44:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:44:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:44:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:44:47,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68a9e740]) (logid:2f0471b4) No inactive management server node found 2025-06-29 03:44:47,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68a9e740]) (logid:2f0471b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:48,066 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-22fdcc70]) (logid:2beeed81) ClusterDRS.poll is being called at 2025-06-29T01:45:00+0000 2025-06-29 03:44:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-22fdcc70]) (logid:2beeed81) Removed 0 old drs migration plans 2025-06-29 03:44:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-af4132df]) (logid:894bec07) VM scheduler.poll is being called at 2025-06-29T01:45:00+0000 2025-06-29 03:44:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-af4132df]) (logid:894bec07) Got 0 scheduled jobs to be executed at 2025-06-29T01:45:00+0000 2025-06-29 03:44:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-af4132df]) (logid:894bec07) Cleaned up 0 VM scheduled job entries 2025-06-29 03:44:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-a480f59d]) (logid:de12b01b) VM Operation Thread Running 2025-06-29 03:44:48,205 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-687aa3e9]) (logid:e2297c41) HA health check task is running... 2025-06-29 03:44:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c60792d2]) (logid:2bad539f) No inactive management server node found 2025-06-29 03:44:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c60792d2]) (logid:2bad539f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2946d308]) (logid:3cd06363) No inactive management server node found 2025-06-29 03:44:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2946d308]) (logid:3cd06363) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:52,206 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7443a604]) (logid:e31b88d9) HA health check task is running... 2025-06-29 03:44:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae1fd8bb]) (logid:8637c31d) No inactive management server node found 2025-06-29 03:44:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae1fd8bb]) (logid:8637c31d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-151e042a]) (logid:300f0f53) No inactive management server node found 2025-06-29 03:44:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-151e042a]) (logid:300f0f53) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:55,230 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8303: Seq 7-8303: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:44:55,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8303 2025-06-29 03:44:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-651078f2]) (logid:d6c8267a) No inactive management server node found 2025-06-29 03:44:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-651078f2]) (logid:d6c8267a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:56,208 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b265f1a5]) (logid:1cfbb095) HA health check task is running... 2025-06-29 03:44:56,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f9788fe]) (logid:20a5abce) No inactive management server node found 2025-06-29 03:44:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f9788fe]) (logid:20a5abce) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:58,067 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-3088bb20]) (logid:90fc1e31) Starting to remove all volume_stats rows older than [Sat Jun 28 15:44:58 CEST 2025]. 2025-06-29 03:44:58,068 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-3088bb20]) (logid:90fc1e31) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:44:58 CEST 2025]. 2025-06-29 03:44:58,069 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-ec84f680]) (logid:99767dd5) AutoScaling Monitor is running... 2025-06-29 03:44:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9da151bb]) (logid:1ed2ada5) Enabled non-edge zones available for scan: 1 2025-06-29 03:44:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9da151bb]) (logid:1ed2ada5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:44:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9da151bb]) (logid:1ed2ada5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:44:58,271 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-2ebd196d]) (logid:1a9746d5) Starting to remove all vm_stats rows older than [Sat Jun 28 15:44:58 CEST 2025]. 2025-06-29 03:44:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6348fbda]) (logid:8712441e) Enabled non-edge zones available for scan: 1 2025-06-29 03:44:58,285 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6348fbda]) (logid:8712441e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:44:58,317 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-2ebd196d]) (logid:1a9746d5) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:44:58 CEST 2025]. 2025-06-29 03:44:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e3c019]) (logid:114103ec) No inactive management server node found 2025-06-29 03:44:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4e3c019]) (logid:114103ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:44:58,543 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:44:58,545 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:44:58,547 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) Seq 2-4725683383995203628: Routed from 90520738888109 2025-06-29 03:44:58,591 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) Seq 1-4725683383995203628: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:44:58,593 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:44:58,594 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) Seq 2-3926857400090361901: Routed from 90520738888109 2025-06-29 03:44:58,639 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d55f9575]) (logid:06f80c73) Seq 2-3926857400090361901: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:44:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bbbe3ed]) (logid:f7d86bd5) No inactive management server node found 2025-06-29 03:44:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bbbe3ed]) (logid:f7d86bd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:00,210 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-16f20309]) (logid:48732712) HA health check task is running... 2025-06-29 03:45:00,369 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:45:00,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:45:00,374 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) Seq 2-4725683383995203629: Routed from 90520738888109 2025-06-29 03:45:00,428 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) Seq 1-4725683383995203629: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:45:00,465 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:45:00,466 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) Seq 2-3926857400090361902: Routed from 90520738888109 2025-06-29 03:45:00,555 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-ef62ed92]) (logid:28be0b1c) Seq 2-3926857400090361902: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:45:00,644 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) StorageCollector is running... 2025-06-29 03:45:00,650 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:45:00,650 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:45:00,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:45:00,651 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3938116399158788112: Routed from 90520738888109 2025-06-29 03:45:00,705 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 4-3938116399158788112: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:45:00,709 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:45:00,709 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:45:00,709 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:45:00,711 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3926857400090361903: Routed from 90520738888109 2025-06-29 03:45:00,723 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3926857400090361903: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:45:00,733 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:45:00,734 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:45:00,734 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:45:00,735 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3926857400090361904: Routed from 90520738888109 2025-06-29 03:45:00,829 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3926857400090361904: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:45:00,839 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:45:00,839 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:45:00,839 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:45:00,840 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-4725683383995203630: Routed from 90520738888109 2025-06-29 03:45:00,884 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 1-4725683383995203630: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:45:00,886 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:45:00,886 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:45:00,886 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:45:00,887 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3926857400090361905: Routed from 90520738888109 2025-06-29 03:45:00,932 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-f6664dde]) (logid:8368eb51) Seq 2-3926857400090361905: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:45:01,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef693e5f]) (logid:e4ea8853) No inactive management server node found 2025-06-29 03:45:01,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef693e5f]) (logid:e4ea8853) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1693ff9e]) (logid:f7cf1461) No inactive management server node found 2025-06-29 03:45:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1693ff9e]) (logid:f7cf1461) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:04,211 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-efd1461b]) (logid:5f6f3894) HA health check task is running... 2025-06-29 03:45:04,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9ed66dc]) (logid:d23d60a7) No inactive management server node found 2025-06-29 03:45:04,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9ed66dc]) (logid:d23d60a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:05,230 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8304: Seq 7-8304: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:45:05,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8304 2025-06-29 03:45:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fceac84e]) (logid:d09cca93) No inactive management server node found 2025-06-29 03:45:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fceac84e]) (logid:d09cca93) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebdec947]) (logid:3283217f) No inactive management server node found 2025-06-29 03:45:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebdec947]) (logid:3283217f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:08,212 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-677eac45]) (logid:997fb416) HA health check task is running... 2025-06-29 03:45:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7c8e3a5]) (logid:8d6c12fb) No inactive management server node found 2025-06-29 03:45:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7c8e3a5]) (logid:8d6c12fb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5ce9323]) (logid:f92d6390) No inactive management server node found 2025-06-29 03:45:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5ce9323]) (logid:f92d6390) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:11,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87b475d2]) (logid:12d80bb7) No inactive management server node found 2025-06-29 03:45:11,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87b475d2]) (logid:12d80bb7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:12,214 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0323caca]) (logid:1253ad4e) HA health check task is running... 2025-06-29 03:45:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-eaf0c1cc]) (logid:b74f7e4d) Found 0 routers to update status. 2025-06-29 03:45:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-eaf0c1cc]) (logid:b74f7e4d) Found 0 VPC's to update Redundant State. 2025-06-29 03:45:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-eaf0c1cc]) (logid:b74f7e4d) Found 0 networks to update RvR status. 2025-06-29 03:45:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b06d542f]) (logid:e15b5489) No inactive management server node found 2025-06-29 03:45:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b06d542f]) (logid:e15b5489) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:14,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c47bf8e3]) (logid:47f3d26a) No inactive management server node found 2025-06-29 03:45:14,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c47bf8e3]) (logid:47f3d26a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:15,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8305: Seq 7-8305: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:45:15,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8305 2025-06-29 03:45:16,216 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4d83f3a4]) (logid:8709dc96) HA health check task is running... 2025-06-29 03:45:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb09557d]) (logid:4d14484e) No inactive management server node found 2025-06-29 03:45:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb09557d]) (logid:4d14484e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bcb8431]) (logid:5881cb09) No inactive management server node found 2025-06-29 03:45:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bcb8431]) (logid:5881cb09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-020a2596]) (logid:f7eaacc8) No inactive management server node found 2025-06-29 03:45:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-020a2596]) (logid:f7eaacc8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:20,217 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-262adbcc]) (logid:f5a7f37d) HA health check task is running... 2025-06-29 03:45:20,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96e0f02d]) (logid:104f2d8d) No inactive management server node found 2025-06-29 03:45:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96e0f02d]) (logid:104f2d8d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:22,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ef6794b]) (logid:5d262196) No inactive management server node found 2025-06-29 03:45:22,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ef6794b]) (logid:5d262196) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3507114]) (logid:6bbbf779) No inactive management server node found 2025-06-29 03:45:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3507114]) (logid:6bbbf779) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:24,219 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-2af1f55a]) (logid:d9fbec4f) HA health check task is running... 2025-06-29 03:45:25,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8306: Seq 7-8306: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:45:25,252 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8306 2025-06-29 03:45:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa5bfd2f]) (logid:c6338617) No inactive management server node found 2025-06-29 03:45:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa5bfd2f]) (logid:c6338617) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f70e6de]) (logid:f7686e6f) No inactive management server node found 2025-06-29 03:45:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f70e6de]) (logid:f7686e6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-32f88569]) (logid:84646d08) Enabled non-edge zones available for scan: 1 2025-06-29 03:45:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-32f88569]) (logid:84646d08) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:45:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-32f88569]) (logid:84646d08) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:45:28,220 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-2f37332e]) (logid:846c46f3) HA health check task is running... 2025-06-29 03:45:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b6e14853]) (logid:03d4ac8c) Enabled non-edge zones available for scan: 1 2025-06-29 03:45:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b6e14853]) (logid:03d4ac8c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:45:28,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-007c83d1]) (logid:9a20b769) No inactive management server node found 2025-06-29 03:45:28,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-007c83d1]) (logid:9a20b769) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7517f29e]) (logid:10b3a39e) No inactive management server node found 2025-06-29 03:45:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7517f29e]) (logid:10b3a39e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:30,232 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8307: Seq 7-8307: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:45:30,253 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8307 2025-06-29 03:45:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1868e7be]) (logid:026b4445) No inactive management server node found 2025-06-29 03:45:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1868e7be]) (logid:026b4445) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:32,222 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-71c71092]) (logid:6376d7c2) HA health check task is running... 2025-06-29 03:45:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e5bb75e]) (logid:7cf98740) No inactive management server node found 2025-06-29 03:45:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e5bb75e]) (logid:7cf98740) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dc4dc30]) (logid:2ab58b0c) No inactive management server node found 2025-06-29 03:45:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4dc4dc30]) (logid:2ab58b0c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3872f32]) (logid:4573c3f6) No inactive management server node found 2025-06-29 03:45:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3872f32]) (logid:4573c3f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:36,223 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a2648378]) (logid:9518bada) HA health check task is running... 2025-06-29 03:45:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-484da7a5]) (logid:8d339613) No inactive management server node found 2025-06-29 03:45:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-484da7a5]) (logid:8d339613) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f80dc197]) (logid:b9956a6c) No inactive management server node found 2025-06-29 03:45:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f80dc197]) (logid:b9956a6c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:40,224 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1ca58f18]) (logid:e0737145) HA health check task is running... 2025-06-29 03:45:40,231 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8308: Seq 7-8308: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:45:40,241 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8308 2025-06-29 03:45:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2ae54bf]) (logid:398c819f) No inactive management server node found 2025-06-29 03:45:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2ae54bf]) (logid:398c819f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0daa02f]) (logid:6fe0c9f4) No inactive management server node found 2025-06-29 03:45:41,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0daa02f]) (logid:6fe0c9f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-d4155958]) (logid:9b1615f7) DbCollector is running... 2025-06-29 03:45:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1128847c]) (logid:548a956d) Found 0 routers to update status. 2025-06-29 03:45:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1128847c]) (logid:548a956d) Found 0 VPC's to update Redundant State. 2025-06-29 03:45:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1128847c]) (logid:548a956d) Found 0 networks to update RvR status. 2025-06-29 03:45:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d77c9b3]) (logid:47262006) No inactive management server node found 2025-06-29 03:45:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d77c9b3]) (logid:47262006) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) ManagementServerCollector is running... 2025-06-29 03:45:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) DB properties were already loaded 2025-06-29 03:45:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:45:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Execution of process [4065184] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:45:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Exit value of process [4065184] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:45:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Process [4065184] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:45:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) usage install: null 2025-06-29 03:45:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Sessions active: 0 2025-06-29 03:45:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:45:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065187] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:45:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:45:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065192] for command [/bin/bash -c uname -r ]. 2025-06-29 03:45:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:45:44,088 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065193] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:45:44,088 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) system memory from /proc: 4887515136 2025-06-29 03:45:44,088 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:45:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065198] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:45:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) free memory from /proc: 185122816 2025-06-29 03:45:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:45:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065203] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:45:44,108 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) used memory from /proc: 1069548 2025-06-29 03:45:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:45:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065204] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:45:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:45:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065205] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:45:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:45:44,132 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065206] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:45:44,133 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:45:44,137 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065213] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:45:44,137 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:45:44,143 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Successfully executed process [4065214] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail buffersdirect.capacity, 216587 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail buffersdirect.count, 72 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail buffersdirect.used, 216587 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail buffersmapped.capacity, 0 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail buffersmapped.count, 0 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail buffersmapped.used, 0 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail jvmuptime, 603849 2025-06-29 03:45:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memoryheap.committed, 545259520 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memoryheap.init, 77594624 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memoryheap.usage, 0.13484868712557505 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorynon-heap.committed, 188022784 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorynon-heap.max, -1 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorynon-heap.usage, -1.86050352E8 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorynon-heap.used, 186050352 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3031798245614035 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09157037290055985 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19812564568267405 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.Compressed-Class-Space.usage, 0.0121697336435318 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.Metaspace.usage, 0.9943295084244353 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.PS-Eden-Space.usage, 0.09793825456158543 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorytotal.committed, 733282304 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorytotal.init, 85262336 2025-06-29 03:45:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorytotal.max, 1908932607 2025-06-29 03:45:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail memorytotal.used, 443782288 2025-06-29 03:45:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail threadsdeadlocks, [] 2025-06-29 03:45:44,148 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail threadsnew.count, 0 2025-06-29 03:45:44,149 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) not storing detail threadstimed_waiting.count, 82 2025-06-29 03:45:44,150 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-3b1acae7]) (logid:a67b1162) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":288758008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":603761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":442538488,"totalCommitted":733282304,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":257417056,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":185122816,"systemMemoryUsed":1069548,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.15,0.18],"systemCyclesUsage":[66397900,53822485,4588851056],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:45:44,151 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 19, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":288758008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":603761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":442538488,"totalCommitted":733282304,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":257417056,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":185122816,"systemMemoryUsed":1069548,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.15,0.18],"systemCyclesUsage":[66397900,53822485,4588851056],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:45:44,151 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:45:44,160 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:45:44,160 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 19, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":288758008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":603761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":442538488,"totalCommitted":733282304,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":257417056,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":185122816,"systemMemoryUsed":1069548,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.15,0.18],"systemCyclesUsage":[66397900,53822485,4588851056],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:45:44,161 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-9:[ctx-016dfeb9]) (logid:feca00e2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":288758008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":603761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":442538488,"totalCommitted":733282304,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":257417056,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":185122816,"systemMemoryUsed":1069548,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.15,0.18],"systemCyclesUsage":[66397900,53822485,4588851056],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:45:44,226 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6a0a5200]) (logid:e56f07b0) HA health check task is running... 2025-06-29 03:45:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5441ef0c]) (logid:2d2bf61f) No inactive management server node found 2025-06-29 03:45:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5441ef0c]) (logid:2d2bf61f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:45:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:45:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48a247e1]) (logid:41404b3d) No inactive management server node found 2025-06-29 03:45:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48a247e1]) (logid:41404b3d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:45:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:45:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:45:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:45:47,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:45:47,088 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:45:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:45:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:45:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cce6f869]) (logid:53f58f9d) No inactive management server node found 2025-06-29 03:45:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cce6f869]) (logid:53f58f9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3f655d09]) (logid:96ca84f1) ClusterDRS.poll is being called at 2025-06-29T01:46:00+0000 2025-06-29 03:45:48,073 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3f655d09]) (logid:96ca84f1) Removed 0 old drs migration plans 2025-06-29 03:45:48,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a0ad4ca3]) (logid:357bdee4) Resetting hosts suitable for reconnect 2025-06-29 03:45:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a0ad4ca3]) (logid:357bdee4) Completed resetting hosts suitable for reconnect 2025-06-29 03:45:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a0ad4ca3]) (logid:357bdee4) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:45:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a0ad4ca3]) (logid:357bdee4) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:45:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a0ad4ca3]) (logid:357bdee4) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:45:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a0ad4ca3]) (logid:357bdee4) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:45:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b0ed6044]) (logid:30f9d6e0) VM scheduler.poll is being called at 2025-06-29T01:46:00+0000 2025-06-29 03:45:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b0ed6044]) (logid:30f9d6e0) Got 0 scheduled jobs to be executed at 2025-06-29T01:46:00+0000 2025-06-29 03:45:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b0ed6044]) (logid:30f9d6e0) Cleaned up 0 VM scheduled job entries 2025-06-29 03:45:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-7a9f397e]) (logid:6775314d) VM Operation Thread Running 2025-06-29 03:45:48,227 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b5fda782]) (logid:208cf5ce) HA health check task is running... 2025-06-29 03:45:49,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d1de701]) (logid:cd542992) No inactive management server node found 2025-06-29 03:45:49,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d1de701]) (logid:cd542992) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:50,233 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8310: Seq 7-8310: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:45:50,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8310 2025-06-29 03:45:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2df26f21]) (logid:5173567b) No inactive management server node found 2025-06-29 03:45:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2df26f21]) (logid:5173567b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:52,228 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-144f16b3]) (logid:1a840d7d) HA health check task is running... 2025-06-29 03:45:52,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ecff880]) (logid:2f5168b7) No inactive management server node found 2025-06-29 03:45:52,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ecff880]) (logid:2f5168b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2a3676b]) (logid:20c8495b) No inactive management server node found 2025-06-29 03:45:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2a3676b]) (logid:20c8495b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:55,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-731fffd4]) (logid:e56d596e) No inactive management server node found 2025-06-29 03:45:55,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-731fffd4]) (logid:e56d596e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:56,230 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5c38bde7]) (logid:4abe3775) HA health check task is running... 2025-06-29 03:45:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9beaafac]) (logid:54d1ef29) No inactive management server node found 2025-06-29 03:45:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9beaafac]) (logid:54d1ef29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:58,070 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-5ca213c3]) (logid:fea0160c) Starting to remove all volume_stats rows older than [Sat Jun 28 15:45:58 CEST 2025]. 2025-06-29 03:45:58,071 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-fa93a96a]) (logid:98227fc6) AutoScaling Monitor is running... 2025-06-29 03:45:58,071 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-5ca213c3]) (logid:fea0160c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:45:58 CEST 2025]. 2025-06-29 03:45:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3960bccc]) (logid:cb7e1866) Enabled non-edge zones available for scan: 1 2025-06-29 03:45:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3960bccc]) (logid:cb7e1866) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2354 TB], threshold=[90.00%]. 2025-06-29 03:45:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3960bccc]) (logid:cb7e1866) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:45:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-abbc27f2]) (logid:f816a031) Enabled non-edge zones available for scan: 1 2025-06-29 03:45:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-abbc27f2]) (logid:f816a031) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:45:58,319 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-b4172e5e]) (logid:37b7f60f) Starting to remove all vm_stats rows older than [Sat Jun 28 15:45:58 CEST 2025]. 2025-06-29 03:45:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef60b288]) (logid:d3b2cf00) No inactive management server node found 2025-06-29 03:45:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef60b288]) (logid:d3b2cf00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:45:58,371 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-b4172e5e]) (logid:37b7f60f) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:45:58 CEST 2025]. 2025-06-29 03:45:58,642 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:45:58,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:45:58,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) Seq 2-4725683383995203631: Routed from 90520738888109 2025-06-29 03:45:58,690 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) Seq 1-4725683383995203631: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:45:58,692 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:45:58,693 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) Seq 2-3926857400090361906: Routed from 90520738888109 2025-06-29 03:45:58,738 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-67ceaa2d]) (logid:d2b11a6c) Seq 2-3926857400090361906: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:45:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7399f4c]) (logid:d1e843a2) No inactive management server node found 2025-06-29 03:45:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7399f4c]) (logid:d1e843a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:00,231 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-74f9bf1f]) (logid:db3ffb5c) HA health check task is running... 2025-06-29 03:46:00,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8311: Seq 7-8311: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:00,258 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8311 2025-06-29 03:46:00,611 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:46:00,616 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:46:00,617 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) Seq 2-4725683383995203632: Routed from 90520738888109 2025-06-29 03:46:00,672 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) Seq 1-4725683383995203632: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:46:00,719 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:46:00,720 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) Seq 2-3926857400090361907: Routed from 90520738888109 2025-06-29 03:46:00,817 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c6f0cbc5]) (logid:0119c304) Seq 2-3926857400090361907: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:46:00,932 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) StorageCollector is running... 2025-06-29 03:46:00,938 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:46:00,938 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:46:00,938 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:46:00,939 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-3938116399158788113: Routed from 90520738888109 2025-06-29 03:46:00,995 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 4-3938116399158788113: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:46:00,999 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:46:00,999 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:46:00,999 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:46:01,001 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-3926857400090361908: Routed from 90520738888109 2025-06-29 03:46:01,027 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-3926857400090361908: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:46:01,038 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:46:01,038 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:46:01,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:46:01,040 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-4725683383995203633: Routed from 90520738888109 2025-06-29 03:46:01,090 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 1-4725683383995203633: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:46:01,101 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:46:01,101 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:46:01,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:46:01,102 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-4725683383995203634: Routed from 90520738888109 2025-06-29 03:46:01,186 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 1-4725683383995203634: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:46:01,189 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:46:01,189 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:46:01,189 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:46:01,190 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-3926857400090361909: Routed from 90520738888109 2025-06-29 03:46:01,236 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-69def186]) (logid:684a6a4a) Seq 2-3926857400090361909: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:46:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-492ede17]) (logid:25a9e751) No inactive management server node found 2025-06-29 03:46:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-492ede17]) (logid:25a9e751) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:02,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed5a65d6]) (logid:c5c2a2ce) No inactive management server node found 2025-06-29 03:46:02,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed5a65d6]) (logid:c5c2a2ce) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:04,233 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2cce92ff]) (logid:3794a3eb) HA health check task is running... 2025-06-29 03:46:04,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a84776b1]) (logid:0bd056f1) No inactive management server node found 2025-06-29 03:46:04,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a84776b1]) (logid:0bd056f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:05,863 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f6b4b72]) (logid:c0a1f20a) No inactive management server node found 2025-06-29 03:46:05,863 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f6b4b72]) (logid:c0a1f20a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:07,366 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ceae0a3]) (logid:3df6a28d) No inactive management server node found 2025-06-29 03:46:07,367 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ceae0a3]) (logid:3df6a28d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:08,236 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bd093077]) (logid:457ad601) HA health check task is running... 2025-06-29 03:46:08,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9488a3e6]) (logid:fc5cd834) No inactive management server node found 2025-06-29 03:46:08,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9488a3e6]) (logid:fc5cd834) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:10,234 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8312: Seq 7-8312: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:10,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8312 2025-06-29 03:46:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1cd703f]) (logid:2c34e3d3) No inactive management server node found 2025-06-29 03:46:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1cd703f]) (logid:2c34e3d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3b405f5]) (logid:5a3676d6) No inactive management server node found 2025-06-29 03:46:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3b405f5]) (logid:5a3676d6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:12,238 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f5000e28]) (logid:f7809906) HA health check task is running... 2025-06-29 03:46:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-99b0c26d]) (logid:56c69ff4) Found 0 routers to update status. 2025-06-29 03:46:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-99b0c26d]) (logid:56c69ff4) Found 0 VPC's to update Redundant State. 2025-06-29 03:46:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-99b0c26d]) (logid:56c69ff4) Found 0 networks to update RvR status. 2025-06-29 03:46:13,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38184747]) (logid:bfa1d14b) No inactive management server node found 2025-06-29 03:46:13,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38184747]) (logid:bfa1d14b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:14,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8369f93d]) (logid:bb1ee956) No inactive management server node found 2025-06-29 03:46:14,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8369f93d]) (logid:bb1ee956) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:15,234 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8313: Seq 7-8313: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:15,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8313 2025-06-29 03:46:16,239 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3b6b2610]) (logid:12a517bb) HA health check task is running... 2025-06-29 03:46:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aa4aa7c]) (logid:507c6131) No inactive management server node found 2025-06-29 03:46:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aa4aa7c]) (logid:507c6131) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:17,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ad57ff2]) (logid:78c83fd3) No inactive management server node found 2025-06-29 03:46:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ad57ff2]) (logid:78c83fd3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:19,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03f15b12]) (logid:6123bf0f) No inactive management server node found 2025-06-29 03:46:19,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03f15b12]) (logid:6123bf0f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:20,242 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-736b9487]) (logid:37abf794) HA health check task is running... 2025-06-29 03:46:20,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaaf5b76]) (logid:7e0cace6) No inactive management server node found 2025-06-29 03:46:20,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaaf5b76]) (logid:7e0cace6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:22,368 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37d427e0]) (logid:3f018c85) No inactive management server node found 2025-06-29 03:46:22,368 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37d427e0]) (logid:3f018c85) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:23,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4520b6c]) (logid:e0efb2b1) No inactive management server node found 2025-06-29 03:46:23,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4520b6c]) (logid:e0efb2b1) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:24,243 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9c85e614]) (logid:6aa118e5) HA health check task is running... 2025-06-29 03:46:25,234 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8314: Seq 7-8314: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:25,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8314 2025-06-29 03:46:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6d7bb29]) (logid:b9a56077) No inactive management server node found 2025-06-29 03:46:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6d7bb29]) (logid:b9a56077) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:26,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5359625]) (logid:609d50ba) No inactive management server node found 2025-06-29 03:46:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5359625]) (logid:609d50ba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4d9cc1a0]) (logid:d8002735) Enabled non-edge zones available for scan: 1 2025-06-29 03:46:28,099 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4d9cc1a0]) (logid:d8002735) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:46:28,102 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4d9cc1a0]) (logid:d8002735) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:46:28,245 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-964d050e]) (logid:cf6de689) HA health check task is running... 2025-06-29 03:46:28,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-62e37ccc]) (logid:2b3e2fea) Enabled non-edge zones available for scan: 1 2025-06-29 03:46:28,284 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-62e37ccc]) (logid:2b3e2fea) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:46:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a3d9184]) (logid:7a797d9d) No inactive management server node found 2025-06-29 03:46:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a3d9184]) (logid:7a797d9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52aac14e]) (logid:50b70795) No inactive management server node found 2025-06-29 03:46:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52aac14e]) (logid:50b70795) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:31,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c35ffa5]) (logid:ffedd808) No inactive management server node found 2025-06-29 03:46:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c35ffa5]) (logid:ffedd808) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:32,246 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-fdff44b2]) (logid:bd6b37d7) HA health check task is running... 2025-06-29 03:46:32,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8255bb13]) (logid:e865951c) No inactive management server node found 2025-06-29 03:46:32,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8255bb13]) (logid:e865951c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:34,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d39ccee]) (logid:72f1c6a6) No inactive management server node found 2025-06-29 03:46:34,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d39ccee]) (logid:72f1c6a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:35,234 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8315: Seq 7-8315: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:35,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8315 2025-06-29 03:46:35,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8466d91]) (logid:d93efa10) No inactive management server node found 2025-06-29 03:46:35,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8466d91]) (logid:d93efa10) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:36,248 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d33bc78c]) (logid:cc09681a) HA health check task is running... 2025-06-29 03:46:37,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5cc067c]) (logid:bf5ef2e7) No inactive management server node found 2025-06-29 03:46:37,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5cc067c]) (logid:bf5ef2e7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67378954]) (logid:b6620047) No inactive management server node found 2025-06-29 03:46:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67378954]) (logid:b6620047) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:40,249 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3e612796]) (logid:9647ecf6) HA health check task is running... 2025-06-29 03:46:40,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43656dbc]) (logid:2d6f91b1) No inactive management server node found 2025-06-29 03:46:40,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43656dbc]) (logid:2d6f91b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:41,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-511da63f]) (logid:52885cf0) No inactive management server node found 2025-06-29 03:46:41,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-511da63f]) (logid:52885cf0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:43,001 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-cdfb82ef]) (logid:69da3432) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 03:46:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-2febaed1]) (logid:bebc1e6d) DbCollector is running... 2025-06-29 03:46:43,092 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-5014d11b]) (logid:05993ae5) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:46:43,094 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-5014d11b]) (logid:05993ae5) Snapshot scheduler is being called at [2025-06-29T01:46:43+0000]. 2025-06-29 03:46:43,094 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-5014d11b]) (logid:05993ae5) There are [0] scheduled snapshots to be executed at [2025-06-29T01:46:43+0000]. 2025-06-29 03:46:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-a4549032]) (logid:3dc2f410) Found 0 running routers. 2025-06-29 03:46:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1991faee]) (logid:21dda387) Found 0 routers to update status. 2025-06-29 03:46:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1991faee]) (logid:21dda387) Found 0 VPC's to update Redundant State. 2025-06-29 03:46:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1991faee]) (logid:21dda387) Found 0 networks to update RvR status. 2025-06-29 03:46:43,111 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Found 1 running routers. 2025-06-29 03:46:43,118 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-a5323d4e]) (logid:c91706da) External devices are not used. Skipping external device usage collection 2025-06-29 03:46:43,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:46:43,122 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Seq 2-3926857400090361910: Routed from 90520738888109 2025-06-29 03:46:43,124 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Seq 2-3926857400090361910: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:43,124 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-0ff9db7a]) (logid:7f2fa989) Started resource counters recalculation periodic task. 2025-06-29 03:46:43,127 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-0ff9db7a]) (logid:7f2fa989) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:46:43,305 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-f0af6b67]) (logid:b1f5997c) Backup backup.poll is being called at 2025-06-29T01:46:43+0000 2025-06-29 03:46:43,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92bcb34d]) (logid:38f57d9c) No inactive management server node found 2025-06-29 03:46:43,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92bcb34d]) (logid:38f57d9c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:43,634 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-0ff9db7a]) (logid:7f2fa989) Finished resource counters recalculation periodic task. 2025-06-29 03:46:43,657 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 2-3926857400090361910: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:43,658 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Seq 2-3926857400090361910: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:46:43,658 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 03:46:43,661 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 03:46:43,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 03:46:43,729 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Seq 2-3926857400090361911: Routed from 90520738888109 2025-06-29 03:46:43,731 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Seq 2-3926857400090361911: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) ManagementServerCollector is running... 2025-06-29 03:46:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) DB properties were already loaded 2025-06-29 03:46:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:46:44,061 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Execution of process [4065253] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:46:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Exit value of process [4065253] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:46:44,062 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Process [4065253] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:46:44,063 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) usage install: null 2025-06-29 03:46:44,063 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Sessions active: 0 2025-06-29 03:46:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:46:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065256] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:46:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:46:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065261] for command [/bin/bash -c uname -r ]. 2025-06-29 03:46:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:46:44,088 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065262] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:46:44,088 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) system memory from /proc: 4887515136 2025-06-29 03:46:44,088 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:46:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065267] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:46:44,097 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) free memory from /proc: 183050240 2025-06-29 03:46:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:46:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065272] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:46:44,110 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) used memory from /proc: 1070316 2025-06-29 03:46:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:46:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065273] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:46:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:46:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065274] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:46:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:46:44,136 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065275] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:46:44,136 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:46:44,141 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065282] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:46:44,141 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:46:44,147 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Successfully executed process [4065283] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail buffersdirect.capacity, 231457 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail buffersdirect.count, 73 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail buffersdirect.used, 231457 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail buffersmapped.capacity, 0 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail buffersmapped.count, 0 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail buffersmapped.used, 0 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:46:44,150 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail jvmuptime, 663854 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memoryheap.committed, 545259520 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memoryheap.init, 77594624 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memoryheap.usage, 0.15454748206595673 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorynon-heap.committed, 188809216 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorynon-heap.max, -1 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorynon-heap.usage, -1.86755648E8 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorynon-heap.used, 186755648 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3031798245614035 2025-06-29 03:46:44,151 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09223477905891762 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20192892491751924 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.Compressed-Class-Space.usage, 0.012182854115962982 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.Metaspace.usage, 0.9939384826543006 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.PS-Eden-Space.usage, 0.15657199441354724 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorytotal.committed, 734068736 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorytotal.init, 85262336 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorytotal.max, 1908932607 2025-06-29 03:46:44,152 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail memorytotal.used, 481777704 2025-06-29 03:46:44,153 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail threadsdeadlocks, [] 2025-06-29 03:46:44,154 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail threadsnew.count, 0 2025-06-29 03:46:44,156 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:46:44,159 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-2bae11dd]) (logid:d74e7083) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":251160192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":663766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":480844464,"totalCommitted":734068736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295020728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":183050240,"systemMemoryUsed":1070316,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.24,0.21],"systemCyclesUsage":[66398343,53822855,4588873969],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:46:44,160 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 21, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":251160192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":663766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":480844464,"totalCommitted":734068736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295020728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":183050240,"systemMemoryUsed":1070316,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.24,0.21],"systemCyclesUsage":[66398343,53822855,4588873969],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:46:44,160 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:46:44,176 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 13 ms 2025-06-29 03:46:44,176 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 15ms. agent: 0, pdu seq: 21, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":251160192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":663766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":480844464,"totalCommitted":734068736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295020728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":183050240,"systemMemoryUsed":1070316,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.24,0.21],"systemCyclesUsage":[66398343,53822855,4588873969],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:46:44,176 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-10:[ctx-03edf114]) (logid:8f200d1f) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":251160192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":663766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":480844464,"totalCommitted":734068736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295020728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":183050240,"systemMemoryUsed":1070316,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.24,0.21],"systemCyclesUsage":[66398343,53822855,4588873969],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:46:44,251 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7318c94c]) (logid:0fb62bc7) HA health check task is running... 2025-06-29 03:46:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-126e4144]) (logid:048b6638) No inactive management server node found 2025-06-29 03:46:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-126e4144]) (logid:048b6638) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:45,114 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 2-3926857400090361911: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.3033da2d-1cd1-44a6-b607-381cddd6d12a","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:45,115 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-9cd463fa]) (logid:b7ff495a) Seq 2-3926857400090361911: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 03:46:45,118 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 03:46:45,122 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:46:45,123 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Seq 2-3926857400090361912: Routed from 90520738888109 2025-06-29 03:46:45,124 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Seq 2-3926857400090361912: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:45,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8316: Seq 7-8316: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:45,256 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8316 2025-06-29 03:46:45,563 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-3926857400090361912: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "13.124465942382812", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "22.984027862548828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "30.792713165283203", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "43.18976402282715", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751161501499", "lastRunDuration": "41.643381118774414", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751161501564", "lastRunDuration": "8075.889825820923", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751161509693", "lastRunDuration": "77.00896263122559", "message": "Memory Usage within limits with current at 50.845%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751161509770", "lastRunDuration": "3251.873254776001", "message": "CPU Usage within limits with current at 0.9%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751161513086", "lastRunDuration": "63.13443183898926", "message": "Sufficient free space is 2571.421875 MB"}, "lastRun": {"start": "2025-06-29 01:45:01.350271", "end": "2025-06-29 01:45:13.150144", "duration": "11.799872636795044"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161201951", "lastRunDuration": "45.058250427246094", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161201996", "lastRunDuration": "60.07218360900879", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161202182", "lastRunDuration": "43.677568435668945", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161202336", "lastRunDuration": "48.874855041503906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:40:01.948981", "end": "2025-06-29 01:40:02.434165", "duration": "0.4851846694946289"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "13.124465942382812", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "22.984027862548828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "30.792713165283203", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "43.18976402282715", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751161501499", "lastRunDuration": "41.643381118774414", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751161501564", "lastRunDuration": "8075.889825820923", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751161509693", "lastRunDuration": "77.00896263122559", "message": "Memory Usage within limits with current at 50.845%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751161509770", "lastRunDuration": "3251.873254776001", "message": "CPU Usage within limits with current at 0.9%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751161513086", "lastRunDuration": "63.13443183898926", "message": "Sufficient free space is 2571.421875 MB"}, "lastRun": {"start": "2025-06-29 01:45:01.350271", "end": "2025-06-29 01:45:13.150144", "duration": "11.799872636795044"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161201951", "lastRunDuration": "45.058250427246094", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161201996", "lastRunDuration": "60.07218360900879", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161202182", "lastRunDuration": "43.677568435668945", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161202336", "lastRunDuration": "48.874855041503906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:40:01.948981", "end": "2025-06-29 01:40:02.434165", "duration": "0.4851846694946289"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:45,564 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Seq 2-3926857400090361912: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:46:45,564 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "13.124465942382812", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "22.984027862548828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "30.792713165283203", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "43.18976402282715", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751161501499", "lastRunDuration": "41.643381118774414", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751161501564", "lastRunDuration": "8075.889825820923", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751161509693", "lastRunDuration": "77.00896263122559", "message": "Memory Usage within limits with current at 50.845%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751161509770", "lastRunDuration": "3251.873254776001", "message": "CPU Usage within limits with current at 0.9%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751161513086", "lastRunDuration": "63.13443183898926", "message": "Sufficient free space is 2571.421875 MB"}, "lastRun": {"start": "2025-06-29 01:45:01.350271", "end": "2025-06-29 01:45:13.150144", "duration": "11.799872636795044"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161201951", "lastRunDuration": "45.058250427246094", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161201996", "lastRunDuration": "60.07218360900879", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161202182", "lastRunDuration": "43.677568435668945", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161202336", "lastRunDuration": "48.874855041503906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:40:01.948981", "end": "2025-06-29 01:40:02.434165", "duration": "0.4851846694946289"}}} 2025-06-29 03:46:45,609 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "13.124465942382812", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "22.984027862548828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "30.792713165283203", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751161501352", "lastRunDuration": "43.18976402282715", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751161501499", "lastRunDuration": "41.643381118774414", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751161501564", "lastRunDuration": "8075.889825820923", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751161509693", "lastRunDuration": "77.00896263122559", "message": "Memory Usage within limits with current at 50.845%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751161509770", "lastRunDuration": "3251.873254776001", "message": "CPU Usage within limits with current at 0.9%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751161513086", "lastRunDuration": "63.13443183898926", "message": "Sufficient free space is 2571.421875 MB"}, "lastRun": {"start": "2025-06-29 01:45:01.350271", "end": "2025-06-29 01:45:13.150144", "duration": "11.799872636795044"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161201951", "lastRunDuration": "45.058250427246094", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161201996", "lastRunDuration": "60.07218360900879", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161202182", "lastRunDuration": "43.677568435668945", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161202336", "lastRunDuration": "48.874855041503906", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:40:01.948981", "end": "2025-06-29 01:40:02.434165", "duration": "0.4851846694946289"}}} 2025-06-29 03:46:45,613 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found check types executed on VR basic, start: 2025-06-29 01:45:01.350271, end: 2025-06-29 01:45:13.150144, duration: 11.799872636795044 2025-06-29 03:46:45,633 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@3b1dbc30- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 03:45:01 CEST 2025, details: service is running which took running duration (ms) 13.124465942382812 2025-06-29 03:46:45,643 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7a7ee2ba- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 03:45:01 CEST 2025, details: service is running which took running duration (ms) 22.984027862548828 2025-06-29 03:46:45,654 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5d139ee2- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 03:45:01 CEST 2025, details: service is running which took running duration (ms) 30.792713165283203 2025-06-29 03:46:45,666 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@78c8b214- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 03:45:01 CEST 2025, details: service is running which took running duration (ms) 43.18976402282715 2025-06-29 03:46:45,677 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@210f29c2- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 03:45:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 41.643381118774414 2025-06-29 03:46:45,688 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2c69646b- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 03:45:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8075.889825820923 2025-06-29 03:46:45,699 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6dcc1b8a- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 03:45:09 CEST 2025, details: Memory Usage within limits with current at 50.845% which took running duration (ms) 77.00896263122559 2025-06-29 03:46:45,710 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@50294c35- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 03:45:09 CEST 2025, details: CPU Usage within limits with current at 0.9% which took running duration (ms) 3251.873254776001 2025-06-29 03:46:45,720 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@8b03ecb- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 03:45:13 CEST 2025, details: Sufficient free space is 2571.421875 MB which took running duration (ms) 63.13443183898926 2025-06-29 03:46:45,720 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found check types executed on VR advanced, start: 2025-06-29 01:40:01.948981, end: 2025-06-29 01:40:02.434165, duration: 0.4851846694946289 2025-06-29 03:46:45,732 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6f1d598a- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 03:40:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 45.058250427246094 2025-06-29 03:46:45,742 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@62800fa8- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 03:40:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 60.07218360900879 2025-06-29 03:46:45,753 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@78d2bf29- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 03:40:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 43.677568435668945 2025-06-29 03:46:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:46:45,764 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-be423dcf]) (logid:11f0aee5) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@719d809e- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 03:40:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 48.874855041503906 2025-06-29 03:46:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:46:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-211a56e5]) (logid:0272b4e2) No inactive management server node found 2025-06-29 03:46:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-211a56e5]) (logid:0272b4e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:46:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:46:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:46:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:46:47,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:46:47,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:46:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:46:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:46:47,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-003f12f8]) (logid:898af6d6) No inactive management server node found 2025-06-29 03:46:47,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-003f12f8]) (logid:898af6d6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:48,037 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-6:[ctx-e9414f75]) (logid:fc9b68d7) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:46:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7e36ba9c]) (logid:2808d677) ClusterDRS.poll is being called at 2025-06-29T01:47:00+0000 2025-06-29 03:46:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7e36ba9c]) (logid:2808d677) Removed 0 old drs migration plans 2025-06-29 03:46:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-63574a17]) (logid:1f0f99ac) VM scheduler.poll is being called at 2025-06-29T01:47:00+0000 2025-06-29 03:46:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-63574a17]) (logid:1f0f99ac) Got 0 scheduled jobs to be executed at 2025-06-29T01:47:00+0000 2025-06-29 03:46:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-63574a17]) (logid:1f0f99ac) Cleaned up 0 VM scheduled job entries 2025-06-29 03:46:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-470b9693]) (logid:4b0b639a) VM Operation Thread Running 2025-06-29 03:46:48,252 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-52e0a4cd]) (logid:54a868f4) HA health check task is running... 2025-06-29 03:46:49,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81dfaf31]) (logid:995e042a) No inactive management server node found 2025-06-29 03:46:49,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81dfaf31]) (logid:995e042a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:46:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27339ce6]) (logid:24411299) No inactive management server node found 2025-06-29 03:46:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27339ce6]) (logid:24411299) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:52,254 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7658d297]) (logid:676e3fec) HA health check task is running... 2025-06-29 03:46:52,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6dab393]) (logid:389ecb6f) No inactive management server node found 2025-06-29 03:46:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6dab393]) (logid:389ecb6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:53,150 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-4861d136]) (logid:61b412f2) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:46:53,162 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-4861d136]) (logid:61b412f2) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:46:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33edba1c]) (logid:1b5a5419) No inactive management server node found 2025-06-29 03:46:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33edba1c]) (logid:1b5a5419) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:55,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8318: Seq 7-8318: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:46:55,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8318 2025-06-29 03:46:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15070171]) (logid:23a7a415) No inactive management server node found 2025-06-29 03:46:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15070171]) (logid:23a7a415) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:56,256 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-8208ae0a]) (logid:19cc5fc2) HA health check task is running... 2025-06-29 03:46:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f82159f6]) (logid:ebf389d7) No inactive management server node found 2025-06-29 03:46:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f82159f6]) (logid:ebf389d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:58,073 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-24533e4a]) (logid:f6e2a5ce) AutoScaling Monitor is running... 2025-06-29 03:46:58,073 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-6fb800e1]) (logid:9974c87f) Starting to remove all volume_stats rows older than [Sat Jun 28 15:46:58 CEST 2025]. 2025-06-29 03:46:58,074 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-6fb800e1]) (logid:9974c87f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:46:58 CEST 2025]. 2025-06-29 03:46:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fd93c2c9]) (logid:3431ed9e) Enabled non-edge zones available for scan: 1 2025-06-29 03:46:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-fd93c2c9]) (logid:3431ed9e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:46:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fd93c2c9]) (logid:3431ed9e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:46:58,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:46:58,115 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Seq 2-4725683383995203635: Routed from 90520738888109 2025-06-29 03:46:58,159 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Seq 1-4725683383995203635: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:46:58,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 03:46:58,193 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:46:58,195 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Seq 2-3926857400090361913: Routed from 90520738888109 2025-06-29 03:46:58,242 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Seq 2-3926857400090361913: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:46:58,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8598b833]) (logid:fb840a78) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 03:46:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-20107ff9]) (logid:75f7a207) Enabled non-edge zones available for scan: 1 2025-06-29 03:46:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-20107ff9]) (logid:75f7a207) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:46:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4b01c38]) (logid:109aeea5) No inactive management server node found 2025-06-29 03:46:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4b01c38]) (logid:109aeea5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:46:58,372 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-d0818eee]) (logid:3969b4be) Starting to remove all vm_stats rows older than [Sat Jun 28 15:46:58 CEST 2025]. 2025-06-29 03:46:58,399 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-d0818eee]) (logid:3969b4be) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:46:58 CEST 2025]. 2025-06-29 03:46:58,742 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:46:58,743 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:46:58,744 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) Seq 2-4725683383995203636: Routed from 90520738888109 2025-06-29 03:46:58,789 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) Seq 1-4725683383995203636: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:46:58,790 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:46:58,791 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) Seq 2-3926857400090361914: Routed from 90520738888109 2025-06-29 03:46:58,836 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-50ec8309]) (logid:40d7e3ab) Seq 2-3926857400090361914: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:46:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66c7c98b]) (logid:d6d77298) No inactive management server node found 2025-06-29 03:46:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66c7c98b]) (logid:d6d77298) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:00,258 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-87fd2389]) (logid:1313359e) HA health check task is running... 2025-06-29 03:47:00,871 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:47:00,876 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:47:00,877 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) Seq 2-4725683383995203637: Routed from 90520738888109 2025-06-29 03:47:00,933 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) Seq 1-4725683383995203637: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:47:00,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:47:00,983 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) Seq 2-3926857400090361915: Routed from 90520738888109 2025-06-29 03:47:01,080 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a38c035c]) (logid:4e421b00) Seq 2-3926857400090361915: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:47:01,237 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) StorageCollector is running... 2025-06-29 03:47:01,246 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:47:01,246 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:47:01,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:47:01,248 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-3938116399158788114: Routed from 90520738888109 2025-06-29 03:47:01,302 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 4-3938116399158788114: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:47:01,309 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:47:01,310 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:47:01,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:47:01,312 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-3926857400090361916: Routed from 90520738888109 2025-06-29 03:47:01,326 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-3926857400090361916: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:47:01,339 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:47:01,339 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:47:01,339 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:47:01,341 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-4725683383995203638: Routed from 90520738888109 2025-06-29 03:47:01,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d095fe5]) (logid:a9723a2d) No inactive management server node found 2025-06-29 03:47:01,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d095fe5]) (logid:a9723a2d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:01,390 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 1-4725683383995203638: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:47:01,401 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:47:01,401 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:47:01,401 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:47:01,403 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-4725683383995203639: Routed from 90520738888109 2025-06-29 03:47:01,489 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 1-4725683383995203639: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:47:01,492 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:47:01,492 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:47:01,492 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:47:01,494 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-3926857400090361917: Routed from 90520738888109 2025-06-29 03:47:01,540 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-74769773]) (logid:caac6646) Seq 2-3926857400090361917: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:47:02,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bacdd776]) (logid:f1692ed4) No inactive management server node found 2025-06-29 03:47:02,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bacdd776]) (logid:f1692ed4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:04,259 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-337dd4bd]) (logid:affc5808) HA health check task is running... 2025-06-29 03:47:04,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc535499]) (logid:43b6b374) No inactive management server node found 2025-06-29 03:47:04,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc535499]) (logid:43b6b374) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:05,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8319: Seq 7-8319: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:47:05,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8319 2025-06-29 03:47:05,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7635a24b]) (logid:4c2e877d) No inactive management server node found 2025-06-29 03:47:05,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7635a24b]) (logid:4c2e877d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:07,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c41e9d97]) (logid:1bea860d) No inactive management server node found 2025-06-29 03:47:07,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c41e9d97]) (logid:1bea860d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:08,261 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d7e9272c]) (logid:f63f9a11) HA health check task is running... 2025-06-29 03:47:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e47478b8]) (logid:6872b1b0) No inactive management server node found 2025-06-29 03:47:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e47478b8]) (logid:6872b1b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:10,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-045f0951]) (logid:b450c0d9) No inactive management server node found 2025-06-29 03:47:10,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-045f0951]) (logid:b450c0d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbb4d254]) (logid:1f5e0350) No inactive management server node found 2025-06-29 03:47:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbb4d254]) (logid:1f5e0350) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:12,263 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-43ab6966]) (logid:96fae397) HA health check task is running... 2025-06-29 03:47:13,002 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-96dec268]) (logid:5e50bac2) Checking if any host reservation can be released ... 2025-06-29 03:47:13,009 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-96dec268]) (logid:5e50bac2) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-96dec268]) (logid:5e50bac2) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:13,016 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-96dec268]) (logid:5e50bac2) Done running HostReservationReleaseChecker ... 2025-06-29 03:47:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9f6f818]) (logid:74388f32) Found 0 routers to update status. 2025-06-29 03:47:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9f6f818]) (logid:74388f32) Found 0 VPC's to update Redundant State. 2025-06-29 03:47:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9f6f818]) (logid:74388f32) Found 0 networks to update RvR status. 2025-06-29 03:47:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Running Capacity Checker ... 2025-06-29 03:47:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) recalculating system capacity 2025-06-29 03:47:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Executing cpu/ram capacity update 2025-06-29 03:47:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:13,173 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:13,192 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:47:13,192 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:47:13,193 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:47:13,207 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:13,208 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:13,218 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:47:13,219 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:47:13,220 DEBUG [c.c.c.CapacityManagerImpl] (pool-16-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:47:13,230 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Done executing cpu/ram capacity update 2025-06-29 03:47:13,230 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Executing storage capacity update 2025-06-29 03:47:13,239 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:47:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:47:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:47:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:47:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:47:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:47:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:47:13,256 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:47:13,263 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:47:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:47:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:47:13,270 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:47:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:47:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:47:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:47:13,283 DEBUG [c.c.s.StorageManagerImpl] (pool-17-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:47:13,284 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Done executing storage capacity update 2025-06-29 03:47:13,284 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Executing capacity updates for public ip and Vlans 2025-06-29 03:47:13,300 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Done capacity updates for public ip and Vlans 2025-06-29 03:47:13,300 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Executing capacity updates for private ip 2025-06-29 03:47:13,304 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Done executing capacity updates for private ip 2025-06-29 03:47:13,304 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Done recalculating system capacity 2025-06-29 03:47:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3c503ae]) (logid:ce2a24de) No inactive management server node found 2025-06-29 03:47:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3c503ae]) (logid:ce2a24de) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:47:13,363 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5fb3dc73]) (logid:487279f2) Done running Capacity Checker ... 2025-06-29 03:47:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-726f8008]) (logid:e086eee2) No inactive management server node found 2025-06-29 03:47:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-726f8008]) (logid:e086eee2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:15,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8320: Seq 7-8320: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:47:15,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8320 2025-06-29 03:47:16,264 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3ddaa0ef]) (logid:68e126f0) HA health check task is running... 2025-06-29 03:47:16,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4bf09d7]) (logid:06c7f966) No inactive management server node found 2025-06-29 03:47:16,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4bf09d7]) (logid:06c7f966) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:17,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b30dcc6]) (logid:2630fae4) No inactive management server node found 2025-06-29 03:47:17,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b30dcc6]) (logid:2630fae4) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-70273dd1]) (logid:16b7cccb) Resetting hosts suitable for reconnect 2025-06-29 03:47:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-70273dd1]) (logid:16b7cccb) Completed resetting hosts suitable for reconnect 2025-06-29 03:47:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-70273dd1]) (logid:16b7cccb) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:47:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-70273dd1]) (logid:16b7cccb) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:47:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-70273dd1]) (logid:16b7cccb) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:47:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-70273dd1]) (logid:16b7cccb) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:47:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f19a5ae4]) (logid:026b5c33) No inactive management server node found 2025-06-29 03:47:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f19a5ae4]) (logid:026b5c33) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:20,266 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-207e4ccf]) (logid:506e4fb2) HA health check task is running... 2025-06-29 03:47:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7121a491]) (logid:4bd1b551) No inactive management server node found 2025-06-29 03:47:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7121a491]) (logid:4bd1b551) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:22,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1020e67]) (logid:57a22fcb) No inactive management server node found 2025-06-29 03:47:22,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1020e67]) (logid:57a22fcb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:23,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbe1d50e]) (logid:05c4d813) No inactive management server node found 2025-06-29 03:47:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbe1d50e]) (logid:05c4d813) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:24,268 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1039dd1f]) (logid:ee530e07) HA health check task is running... 2025-06-29 03:47:25,236 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8321: Seq 7-8321: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:47:25,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8321 2025-06-29 03:47:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5358a12e]) (logid:34b415c6) No inactive management server node found 2025-06-29 03:47:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5358a12e]) (logid:34b415c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:26,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fff30846]) (logid:3276ded5) No inactive management server node found 2025-06-29 03:47:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fff30846]) (logid:3276ded5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-983ba162]) (logid:6ce7c6c3) Enabled non-edge zones available for scan: 1 2025-06-29 03:47:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-983ba162]) (logid:6ce7c6c3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:47:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-983ba162]) (logid:6ce7c6c3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:47:28,269 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-fe2eb871]) (logid:7361781b) HA health check task is running... 2025-06-29 03:47:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e2a1fb1e]) (logid:00518939) Enabled non-edge zones available for scan: 1 2025-06-29 03:47:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e2a1fb1e]) (logid:00518939) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:47:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b88ebf6]) (logid:7de5f0b2) No inactive management server node found 2025-06-29 03:47:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b88ebf6]) (logid:7de5f0b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0668a07]) (logid:ac2bc629) No inactive management server node found 2025-06-29 03:47:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0668a07]) (logid:ac2bc629) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-006bbf96]) (logid:cdd75951) No inactive management server node found 2025-06-29 03:47:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-006bbf96]) (logid:cdd75951) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:32,271 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-74cff0a5]) (logid:f096d41f) HA health check task is running... 2025-06-29 03:47:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c75ff6be]) (logid:51214068) No inactive management server node found 2025-06-29 03:47:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c75ff6be]) (logid:51214068) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2226d735]) (logid:4fc8b956) No inactive management server node found 2025-06-29 03:47:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2226d735]) (logid:4fc8b956) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:35,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8322: Seq 7-8322: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:47:35,258 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8322 2025-06-29 03:47:35,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-934d8bbb]) (logid:c632caf0) No inactive management server node found 2025-06-29 03:47:35,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-934d8bbb]) (logid:c632caf0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:36,272 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ef9e366b]) (logid:0bed72a7) HA health check task is running... 2025-06-29 03:47:37,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-281cc35f]) (logid:aff9b834) No inactive management server node found 2025-06-29 03:47:37,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-281cc35f]) (logid:aff9b834) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98003567]) (logid:694a4a72) No inactive management server node found 2025-06-29 03:47:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98003567]) (logid:694a4a72) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:40,273 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9607998e]) (logid:30f0a74c) HA health check task is running... 2025-06-29 03:47:40,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e7badf8]) (logid:68e7f9e9) No inactive management server node found 2025-06-29 03:47:40,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e7badf8]) (logid:68e7f9e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15d61cd2]) (logid:fa3f601e) No inactive management server node found 2025-06-29 03:47:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15d61cd2]) (logid:fa3f601e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-8274bfc4]) (logid:c0b6b22f) DbCollector is running... 2025-06-29 03:47:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c4744142]) (logid:17799019) Found 0 routers to update status. 2025-06-29 03:47:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c4744142]) (logid:17799019) Found 0 VPC's to update Redundant State. 2025-06-29 03:47:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c4744142]) (logid:17799019) Found 0 networks to update RvR status. 2025-06-29 03:47:43,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-375a7f31]) (logid:093d6825) No inactive management server node found 2025-06-29 03:47:43,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-375a7f31]) (logid:093d6825) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) ManagementServerCollector is running... 2025-06-29 03:47:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) DB properties were already loaded 2025-06-29 03:47:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:47:44,060 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Execution of process [4065291] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:47:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Exit value of process [4065291] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:47:44,060 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Process [4065291] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:47:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) usage install: null 2025-06-29 03:47:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Sessions active: 0 2025-06-29 03:47:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:47:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065294] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:47:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:47:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065299] for command [/bin/bash -c uname -r ]. 2025-06-29 03:47:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:47:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065300] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:47:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) system memory from /proc: 4887515136 2025-06-29 03:47:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:47:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065305] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:47:44,086 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) free memory from /proc: 184643584 2025-06-29 03:47:44,087 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:47:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065310] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:47:44,100 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) used memory from /proc: 1071084 2025-06-29 03:47:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:47:44,106 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065311] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:47:44,106 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:47:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065312] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:47:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:47:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065313] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:47:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:47:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065320] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:47:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:47:44,133 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Successfully executed process [4065321] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail buffersdirect.capacity, 236862 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail buffersdirect.count, 73 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail buffersdirect.used, 236862 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail buffersmapped.capacity, 0 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail buffersmapped.count, 0 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail buffersmapped.used, 0 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:47:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail jvmuptime, 723840 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memoryheap.committed, 545259520 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memoryheap.init, 77594624 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memoryheap.usage, 0.1758762517822735 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorynon-heap.committed, 189595648 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorynon-heap.max, -1 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorynon-heap.usage, -1.87499312E8 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorynon-heap.used, 187499312 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09443106838176486 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20452311127403605 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.Compressed-Class-Space.usage, 0.012196563184261322 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.Metaspace.usage, 0.9939938012156939 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.PS-Eden-Space.usage, 0.22059347053647335 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorytotal.committed, 734855168 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorytotal.init, 85262336 2025-06-29 03:47:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorytotal.max, 1908932607 2025-06-29 03:47:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail memorytotal.used, 523235224 2025-06-29 03:47:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail threadsdeadlocks, [] 2025-06-29 03:47:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail threadsnew.count, 0 2025-06-29 03:47:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:47:44,141 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-4c5110d5]) (logid:5dae3dc7) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":210702320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":723764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.34,"totalInit":85262336,"totalUsed":522016456,"totalCommitted":734855168,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":335735912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":184643584,"systemMemoryUsed":1071084,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.34,0.24,0.2],"systemCyclesUsage":[66398850,53823183,4588896872],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:47:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 23, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":210702320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":723764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.34,"totalInit":85262336,"totalUsed":522016456,"totalCommitted":734855168,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":335735912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":184643584,"systemMemoryUsed":1071084,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.34,0.24,0.2],"systemCyclesUsage":[66398850,53823183,4588896872],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:47:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:47:44,149 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-10:[ctx-12d50f39]) (logid:1cfdcd90) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":210702320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":723764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.34,"totalInit":85262336,"totalUsed":522016456,"totalCommitted":734855168,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":335735912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":184643584,"systemMemoryUsed":1071084,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.34,0.24,0.2],"systemCyclesUsage":[66398850,53823183,4588896872],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:47:44,151 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:47:44,151 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 23, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":210702320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":723764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.34,"totalInit":85262336,"totalUsed":522016456,"totalCommitted":734855168,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":335735912,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":195,"threadsWaitingCount":103,"systemMemoryTotal":4887515136,"systemMemoryFree":184643584,"systemMemoryUsed":1071084,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.34,0.24,0.2],"systemCyclesUsage":[66398850,53823183,4588896872],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:47:44,275 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-858b1b87]) (logid:41e55715) HA health check task is running... 2025-06-29 03:47:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f36bd0]) (logid:6a6bee19) No inactive management server node found 2025-06-29 03:47:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f36bd0]) (logid:6a6bee19) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:45,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8323: Seq 7-8323: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:47:45,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8323 2025-06-29 03:47:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:47:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:47:46,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f42c5cfc]) (logid:21a36f85) No inactive management server node found 2025-06-29 03:47:46,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f42c5cfc]) (logid:21a36f85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:47:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:47:47,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:47:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:47:47,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e761401d]) (logid:c82f0459) No inactive management server node found 2025-06-29 03:47:47,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e761401d]) (logid:c82f0459) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:47:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4e54e886]) (logid:b1567f0f) ClusterDRS.poll is being called at 2025-06-29T01:48:00+0000 2025-06-29 03:47:48,077 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4e54e886]) (logid:b1567f0f) Removed 0 old drs migration plans 2025-06-29 03:47:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2173dae9]) (logid:642cd2fd) VM scheduler.poll is being called at 2025-06-29T01:48:00+0000 2025-06-29 03:47:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2173dae9]) (logid:642cd2fd) Got 0 scheduled jobs to be executed at 2025-06-29T01:48:00+0000 2025-06-29 03:47:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2173dae9]) (logid:642cd2fd) Cleaned up 0 VM scheduled job entries 2025-06-29 03:47:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-104a2586]) (logid:87fd31bc) VM Operation Thread Running 2025-06-29 03:47:48,276 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4b285996]) (logid:749a9ce6) HA health check task is running... 2025-06-29 03:47:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8336c871]) (logid:8b16c9ac) No inactive management server node found 2025-06-29 03:47:49,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8336c871]) (logid:8b16c9ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:50,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8325: Seq 7-8325: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:47:50,260 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8325 2025-06-29 03:47:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd886424]) (logid:f7883833) No inactive management server node found 2025-06-29 03:47:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd886424]) (logid:f7883833) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:52,277 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2ed19e5a]) (logid:93636f9d) HA health check task is running... 2025-06-29 03:47:52,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81a2409b]) (logid:3f69c963) No inactive management server node found 2025-06-29 03:47:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81a2409b]) (logid:3f69c963) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79dfeaa6]) (logid:6248119b) No inactive management server node found 2025-06-29 03:47:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79dfeaa6]) (logid:6248119b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:55,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4eaa98e2]) (logid:dedccf43) No inactive management server node found 2025-06-29 03:47:55,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4eaa98e2]) (logid:dedccf43) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:56,279 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-00fb4195]) (logid:a34b3dcb) HA health check task is running... 2025-06-29 03:47:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca33aa45]) (logid:d571db72) No inactive management server node found 2025-06-29 03:47:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca33aa45]) (logid:d571db72) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:58,075 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-4725a3c9]) (logid:d2735390) AutoScaling Monitor is running... 2025-06-29 03:47:58,077 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-7ef91fd1]) (logid:45be59aa) Starting to remove all volume_stats rows older than [Sat Jun 28 15:47:58 CEST 2025]. 2025-06-29 03:47:58,078 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-7ef91fd1]) (logid:45be59aa) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:47:58 CEST 2025]. 2025-06-29 03:47:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-db199b23]) (logid:ae321605) Enabled non-edge zones available for scan: 1 2025-06-29 03:47:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-db199b23]) (logid:ae321605) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:47:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-db199b23]) (logid:ae321605) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:47:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4cf3f2e1]) (logid:1bbf3b4b) Enabled non-edge zones available for scan: 1 2025-06-29 03:47:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4cf3f2e1]) (logid:1bbf3b4b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:47:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c5919bb]) (logid:c0bc51cc) No inactive management server node found 2025-06-29 03:47:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c5919bb]) (logid:c0bc51cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:47:58,400 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-aa53e46e]) (logid:6618f121) Starting to remove all vm_stats rows older than [Sat Jun 28 15:47:58 CEST 2025]. 2025-06-29 03:47:58,432 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-aa53e46e]) (logid:6618f121) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:47:58 CEST 2025]. 2025-06-29 03:47:58,839 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:47:58,841 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:47:58,842 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) Seq 2-4725683383995203640: Routed from 90520738888109 2025-06-29 03:47:58,887 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) Seq 1-4725683383995203640: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:47:58,888 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:47:58,889 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) Seq 2-3926857400090361918: Routed from 90520738888109 2025-06-29 03:47:58,934 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0eb9ddb2]) (logid:96c49dfe) Seq 2-3926857400090361918: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:47:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8860124f]) (logid:20f7ab2b) No inactive management server node found 2025-06-29 03:47:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8860124f]) (logid:20f7ab2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:00,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8326: Seq 7-8326: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:48:00,258 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8326 2025-06-29 03:48:00,281 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-aaed4758]) (logid:ce929572) HA health check task is running... 2025-06-29 03:48:01,160 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:48:01,163 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:48:01,164 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) Seq 2-4725683383995203641: Routed from 90520738888109 2025-06-29 03:48:01,222 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) Seq 1-4725683383995203641: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:48:01,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:48:01,273 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) Seq 2-3926857400090361919: Routed from 90520738888109 2025-06-29 03:48:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6301dcf2]) (logid:93769cab) No inactive management server node found 2025-06-29 03:48:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6301dcf2]) (logid:93769cab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:01,376 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c6de5aea]) (logid:3350a5da) Seq 2-3926857400090361919: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:48:01,540 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) StorageCollector is running... 2025-06-29 03:48:01,548 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:48:01,548 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:48:01,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:48:01,549 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-3938116399158788115: Routed from 90520738888109 2025-06-29 03:48:01,602 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 4-3938116399158788115: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:48:01,606 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:48:01,606 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:48:01,606 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:48:01,607 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-4725683383995203642: Routed from 90520738888109 2025-06-29 03:48:01,656 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 1-4725683383995203642: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:48:01,680 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:48:01,680 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:48:01,680 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:48:01,681 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-3926857400090361920: Routed from 90520738888109 2025-06-29 03:48:01,693 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-3926857400090361920: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:48:01,705 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:48:01,705 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:48:01,705 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:48:01,706 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-4725683383995203643: Routed from 90520738888109 2025-06-29 03:48:01,750 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 1-4725683383995203643: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:48:01,752 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:48:01,752 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:48:01,752 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:48:01,753 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-3926857400090361921: Routed from 90520738888109 2025-06-29 03:48:01,798 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e90a3433]) (logid:fd80265b) Seq 2-3926857400090361921: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:48:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-331e36f2]) (logid:0c0e37c7) No inactive management server node found 2025-06-29 03:48:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-331e36f2]) (logid:0c0e37c7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:48:04,282 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-825b1265]) (logid:053ce834) HA health check task is running... 2025-06-29 03:48:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bf426fe]) (logid:36595e0b) No inactive management server node found 2025-06-29 03:48:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bf426fe]) (logid:36595e0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6497542]) (logid:06ccbdda) No inactive management server node found 2025-06-29 03:48:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6497542]) (logid:06ccbdda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:07,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5ace7b9]) (logid:99af4228) No inactive management server node found 2025-06-29 03:48:07,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5ace7b9]) (logid:99af4228) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:08,284 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0a53d18c]) (logid:8ea191f6) HA health check task is running... 2025-06-29 03:48:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1c6416c]) (logid:499c5ca3) No inactive management server node found 2025-06-29 03:48:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1c6416c]) (logid:499c5ca3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:10,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8327: Seq 7-8327: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:48:10,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8327 2025-06-29 03:48:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51568a57]) (logid:fe231267) No inactive management server node found 2025-06-29 03:48:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51568a57]) (logid:fe231267) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:11,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bf7791d]) (logid:9380c0c0) No inactive management server node found 2025-06-29 03:48:11,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bf7791d]) (logid:9380c0c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:12,286 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7f18e5f2]) (logid:5598efb7) HA health check task is running... 2025-06-29 03:48:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cd3219ad]) (logid:5dc662b7) Found 0 routers to update status. 2025-06-29 03:48:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cd3219ad]) (logid:5dc662b7) Found 0 VPC's to update Redundant State. 2025-06-29 03:48:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cd3219ad]) (logid:5dc662b7) Found 0 networks to update RvR status. 2025-06-29 03:48:13,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a36d987]) (logid:0efe5f12) No inactive management server node found 2025-06-29 03:48:13,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a36d987]) (logid:0efe5f12) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57e24b3a]) (logid:a4730e55) No inactive management server node found 2025-06-29 03:48:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57e24b3a]) (logid:a4730e55) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:16,287 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0a19be2a]) (logid:bec38318) HA health check task is running... 2025-06-29 03:48:16,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-395ad834]) (logid:3c0cccc8) No inactive management server node found 2025-06-29 03:48:16,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-395ad834]) (logid:3c0cccc8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:17,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b70a9a63]) (logid:c83d930a) No inactive management server node found 2025-06-29 03:48:17,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b70a9a63]) (logid:c83d930a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:48:19,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bc091e0]) (logid:b0113338) No inactive management server node found 2025-06-29 03:48:19,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bc091e0]) (logid:b0113338) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:20,238 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8328: Seq 7-8328: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:48:20,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8328 2025-06-29 03:48:20,289 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6d31fb5a]) (logid:e3946a94) HA health check task is running... 2025-06-29 03:48:20,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d9d4493]) (logid:74c1e87a) No inactive management server node found 2025-06-29 03:48:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d9d4493]) (logid:74c1e87a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:22,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7434bd9c]) (logid:23d5bb60) No inactive management server node found 2025-06-29 03:48:22,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7434bd9c]) (logid:23d5bb60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec9fcefe]) (logid:c9111947) No inactive management server node found 2025-06-29 03:48:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec9fcefe]) (logid:c9111947) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:24,290 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6b5d7e09]) (logid:a66d2908) HA health check task is running... 2025-06-29 03:48:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f93374ca]) (logid:4a24da2d) No inactive management server node found 2025-06-29 03:48:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f93374ca]) (logid:4a24da2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-444755ba]) (logid:919263f4) No inactive management server node found 2025-06-29 03:48:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-444755ba]) (logid:919263f4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6b8f01f9]) (logid:0ea2da61) Enabled non-edge zones available for scan: 1 2025-06-29 03:48:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6b8f01f9]) (logid:0ea2da61) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:48:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6b8f01f9]) (logid:0ea2da61) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:48:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c68bd300]) (logid:ae19ee0a) Enabled non-edge zones available for scan: 1 2025-06-29 03:48:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c68bd300]) (logid:ae19ee0a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:48:28,292 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ff257bcf]) (logid:2966fd21) HA health check task is running... 2025-06-29 03:48:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56701a51]) (logid:839a22c0) No inactive management server node found 2025-06-29 03:48:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56701a51]) (logid:839a22c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:29,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd505c27]) (logid:9b6938ff) No inactive management server node found 2025-06-29 03:48:29,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd505c27]) (logid:9b6938ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:30,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8329: Seq 7-8329: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:48:30,260 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8329 2025-06-29 03:48:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c26c01fb]) (logid:d79d9b15) No inactive management server node found 2025-06-29 03:48:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c26c01fb]) (logid:d79d9b15) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:32,293 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2c9fedbb]) (logid:6f77b1b7) HA health check task is running... 2025-06-29 03:48:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a692f1a]) (logid:fa7437aa) No inactive management server node found 2025-06-29 03:48:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a692f1a]) (logid:fa7437aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59d261fa]) (logid:4a15c236) No inactive management server node found 2025-06-29 03:48:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59d261fa]) (logid:4a15c236) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02c11366]) (logid:27d9be92) No inactive management server node found 2025-06-29 03:48:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02c11366]) (logid:27d9be92) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:36,295 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3e33bad9]) (logid:4bbb7fea) HA health check task is running... 2025-06-29 03:48:37,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c60c5101]) (logid:0459c626) No inactive management server node found 2025-06-29 03:48:37,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c60c5101]) (logid:0459c626) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:38,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-943e6508]) (logid:1396ce2f) No inactive management server node found 2025-06-29 03:48:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-943e6508]) (logid:1396ce2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:40,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8330: Seq 7-8330: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:48:40,261 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8330 2025-06-29 03:48:40,297 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0c290c75]) (logid:f3c4ac42) HA health check task is running... 2025-06-29 03:48:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bc32e23]) (logid:eca111b8) No inactive management server node found 2025-06-29 03:48:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bc32e23]) (logid:eca111b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:41,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1299a3f8]) (logid:1ab3a030) No inactive management server node found 2025-06-29 03:48:41,869 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1299a3f8]) (logid:1ab3a030) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-947686fe]) (logid:dc682424) DbCollector is running... 2025-06-29 03:48:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-17b8bc7b]) (logid:f7e71103) Found 0 routers to update status. 2025-06-29 03:48:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-17b8bc7b]) (logid:f7e71103) Found 0 VPC's to update Redundant State. 2025-06-29 03:48:43,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-17b8bc7b]) (logid:f7e71103) Found 0 networks to update RvR status. 2025-06-29 03:48:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86f762b4]) (logid:658eaebe) No inactive management server node found 2025-06-29 03:48:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86f762b4]) (logid:658eaebe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) ManagementServerCollector is running... 2025-06-29 03:48:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) DB properties were already loaded 2025-06-29 03:48:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:48:44,059 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Execution of process [4065335] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:48:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Exit value of process [4065335] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:48:44,059 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Process [4065335] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:48:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) usage install: null 2025-06-29 03:48:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Sessions active: 0 2025-06-29 03:48:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:48:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065338] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:48:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:48:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065343] for command [/bin/bash -c uname -r ]. 2025-06-29 03:48:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:48:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065344] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:48:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) system memory from /proc: 4887515136 2025-06-29 03:48:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:48:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065349] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:48:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) free memory from /proc: 187150336 2025-06-29 03:48:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:48:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065354] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:48:44,098 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) used memory from /proc: 1071468 2025-06-29 03:48:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:48:44,104 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065355] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:48:44,104 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:48:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065356] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:48:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:48:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065357] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:48:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:48:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065364] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:48:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:48:44,132 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Successfully executed process [4065365] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail buffersdirect.capacity, 236898 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail buffersdirect.count, 73 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail buffersdirect.used, 236898 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail buffersmapped.capacity, 0 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail buffersmapped.count, 0 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail buffersmapped.used, 0 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail jvmuptime, 783838 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memoryheap.committed, 545259520 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memoryheap.init, 77594624 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memoryheap.usage, 0.18692416406142714 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorynon-heap.committed, 189857792 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorynon-heap.max, -1 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorynon-heap.usage, -1.87821384E8 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorynon-heap.used, 187821384 2025-06-29 03:48:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09462060117302053 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2066496984037058 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012196563184261322 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.Metaspace.usage, 0.9942673768201702 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.PS-Eden-Space.usage, 0.25375542322270456 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorytotal.committed, 735117312 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorytotal.init, 85262336 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorytotal.max, 1908932607 2025-06-29 03:48:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail memorytotal.used, 544647016 2025-06-29 03:48:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail threadsdeadlocks, [] 2025-06-29 03:48:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail threadsnew.count, 0 2025-06-29 03:48:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:48:44,139 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-05c92720]) (logid:2661d55b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":189357656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":783762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":543717488,"totalCommitted":735117312,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":356825632,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":187150336,"systemMemoryUsed":1071468,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.19,0.19],"systemCyclesUsage":[66399194,53823495,4588919964],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:48:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 25, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":189357656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":783762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":543717488,"totalCommitted":735117312,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":356825632,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":187150336,"systemMemoryUsed":1071468,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.19,0.19],"systemCyclesUsage":[66399194,53823495,4588919964],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:48:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:48:44,148 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-10:[ctx-e43bf9fd]) (logid:fa6de142) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":189357656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":783762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":543717488,"totalCommitted":735117312,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":356825632,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":187150336,"systemMemoryUsed":1071468,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.19,0.19],"systemCyclesUsage":[66399194,53823495,4588919964],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:48:44,149 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:48:44,149 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 25, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":189357656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":783762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":543717488,"totalCommitted":735117312,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":356825632,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":187150336,"systemMemoryUsed":1071468,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.19,0.19],"systemCyclesUsage":[66399194,53823495,4588919964],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:48:44,298 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-48b0d346]) (logid:436dddb1) HA health check task is running... 2025-06-29 03:48:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8caf6518]) (logid:d5091cac) No inactive management server node found 2025-06-29 03:48:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8caf6518]) (logid:d5091cac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:48:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:48:46,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0116b08d]) (logid:2a613361) No inactive management server node found 2025-06-29 03:48:46,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0116b08d]) (logid:2a613361) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:48:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:48:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:48:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:48:47,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:48:47,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:48:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:48:47,083 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:48:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a51968ae]) (logid:5c4f3974) No inactive management server node found 2025-06-29 03:48:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a51968ae]) (logid:5c4f3974) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3bae55f3]) (logid:821122cf) ClusterDRS.poll is being called at 2025-06-29T01:49:00+0000 2025-06-29 03:48:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3bae55f3]) (logid:821122cf) Removed 0 old drs migration plans 2025-06-29 03:48:48,086 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-688b767f]) (logid:6c9e09f5) Resetting hosts suitable for reconnect 2025-06-29 03:48:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-688b767f]) (logid:6c9e09f5) Completed resetting hosts suitable for reconnect 2025-06-29 03:48:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-688b767f]) (logid:6c9e09f5) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:48:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-688b767f]) (logid:6c9e09f5) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:48:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-688b767f]) (logid:6c9e09f5) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:48:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-688b767f]) (logid:6c9e09f5) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:48:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7c7b1d83]) (logid:18b3fd26) VM scheduler.poll is being called at 2025-06-29T01:49:00+0000 2025-06-29 03:48:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7c7b1d83]) (logid:18b3fd26) Got 0 scheduled jobs to be executed at 2025-06-29T01:49:00+0000 2025-06-29 03:48:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7c7b1d83]) (logid:18b3fd26) Cleaned up 0 VM scheduled job entries 2025-06-29 03:48:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-c717fd6c]) (logid:83b557f3) VM Operation Thread Running 2025-06-29 03:48:48,299 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e1e0da50]) (logid:cf458835) HA health check task is running... 2025-06-29 03:48:49,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42d014f0]) (logid:0dc67c89) No inactive management server node found 2025-06-29 03:48:49,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42d014f0]) (logid:0dc67c89) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:50,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8332: Seq 7-8332: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:48:50,250 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8332 2025-06-29 03:48:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d3280b]) (logid:9e5b0d63) No inactive management server node found 2025-06-29 03:48:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d3280b]) (logid:9e5b0d63) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:52,301 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-87420af6]) (logid:794057e2) HA health check task is running... 2025-06-29 03:48:52,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-048ce585]) (logid:d536b1f3) No inactive management server node found 2025-06-29 03:48:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-048ce585]) (logid:d536b1f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2489eef5]) (logid:e7d376e4) No inactive management server node found 2025-06-29 03:48:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2489eef5]) (logid:e7d376e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:55,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b73be796]) (logid:8f4b699f) No inactive management server node found 2025-06-29 03:48:55,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b73be796]) (logid:8f4b699f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:56,302 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d2b1ed37]) (logid:99d6e0bd) HA health check task is running... 2025-06-29 03:48:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6dad131]) (logid:b22b10de) No inactive management server node found 2025-06-29 03:48:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6dad131]) (logid:b22b10de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:58,078 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-3f6479a9]) (logid:677cdd2f) AutoScaling Monitor is running... 2025-06-29 03:48:58,081 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-4ae1ce0c]) (logid:27a6398c) Starting to remove all volume_stats rows older than [Sat Jun 28 15:48:58 CEST 2025]. 2025-06-29 03:48:58,082 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-4ae1ce0c]) (logid:27a6398c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:48:58 CEST 2025]. 2025-06-29 03:48:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4dc3bc2a]) (logid:4bc88f81) Enabled non-edge zones available for scan: 1 2025-06-29 03:48:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4dc3bc2a]) (logid:4bc88f81) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:48:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4dc3bc2a]) (logid:4bc88f81) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:48:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9b350a8c]) (logid:a13ffbed) Enabled non-edge zones available for scan: 1 2025-06-29 03:48:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9b350a8c]) (logid:a13ffbed) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:48:58,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa263d48]) (logid:3d2e750b) No inactive management server node found 2025-06-29 03:48:58,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa263d48]) (logid:3d2e750b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:48:58,434 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-d91e8693]) (logid:897955f5) Starting to remove all vm_stats rows older than [Sat Jun 28 15:48:58 CEST 2025]. 2025-06-29 03:48:58,457 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-d91e8693]) (logid:897955f5) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:48:58 CEST 2025]. 2025-06-29 03:48:58,937 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:48:58,938 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:48:58,940 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) Seq 2-4725683383995203644: Routed from 90520738888109 2025-06-29 03:48:58,984 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) Seq 1-4725683383995203644: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:48:58,985 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:48:58,986 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) Seq 2-3926857400090361922: Routed from 90520738888109 2025-06-29 03:48:59,031 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-6ee556bf]) (logid:fe4f40c4) Seq 2-3926857400090361922: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:48:59,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74820ead]) (logid:86dd84a2) No inactive management server node found 2025-06-29 03:48:59,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74820ead]) (logid:86dd84a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:00,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8333: Seq 7-8333: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:00,262 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8333 2025-06-29 03:49:00,304 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-84bff99c]) (logid:57eb6f44) HA health check task is running... 2025-06-29 03:49:01,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efbe89b5]) (logid:d3391ce5) No inactive management server node found 2025-06-29 03:49:01,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efbe89b5]) (logid:d3391ce5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:01,433 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:49:01,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:49:01,439 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) Seq 2-4725683383995203645: Routed from 90520738888109 2025-06-29 03:49:01,497 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) Seq 1-4725683383995203645: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:49:01,539 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:49:01,540 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) Seq 2-3926857400090361923: Routed from 90520738888109 2025-06-29 03:49:01,628 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-c1727d72]) (logid:707712bd) Seq 2-3926857400090361923: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:49:01,799 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) StorageCollector is running... 2025-06-29 03:49:01,805 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:49:01,805 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:49:01,805 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:49:01,806 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3938116399158788116: Routed from 90520738888109 2025-06-29 03:49:01,884 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 4-3938116399158788116: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:49:01,888 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:49:01,888 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:49:01,888 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:49:01,889 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3926857400090361924: Routed from 90520738888109 2025-06-29 03:49:01,904 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3926857400090361924: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:49:01,916 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:49:01,916 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:49:01,916 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:49:01,917 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3926857400090361925: Routed from 90520738888109 2025-06-29 03:49:02,017 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3926857400090361925: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:49:02,028 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:49:02,028 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:49:02,028 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:49:02,029 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-4725683383995203646: Routed from 90520738888109 2025-06-29 03:49:02,074 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 1-4725683383995203646: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:49:02,076 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:49:02,076 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:49:02,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:49:02,078 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3926857400090361926: Routed from 90520738888109 2025-06-29 03:49:02,124 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2bdc2de5]) (logid:399da34e) Seq 2-3926857400090361926: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:49:02,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac802982]) (logid:3e32fc22) No inactive management server node found 2025-06-29 03:49:02,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac802982]) (logid:3e32fc22) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:04,305 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9fbe175d]) (logid:5abc3d28) HA health check task is running... 2025-06-29 03:49:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52a7ae5a]) (logid:d8bb766a) No inactive management server node found 2025-06-29 03:49:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52a7ae5a]) (logid:d8bb766a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:05,239 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8334: Seq 7-8334: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:05,261 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8334 2025-06-29 03:49:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88d5a7ac]) (logid:cf642b35) No inactive management server node found 2025-06-29 03:49:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88d5a7ac]) (logid:cf642b35) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28716f62]) (logid:c1c188bd) No inactive management server node found 2025-06-29 03:49:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28716f62]) (logid:c1c188bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:08,307 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6da287e6]) (logid:a6511032) HA health check task is running... 2025-06-29 03:49:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e624a58]) (logid:7ed05070) No inactive management server node found 2025-06-29 03:49:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e624a58]) (logid:7ed05070) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fd06c56]) (logid:8f424ab7) No inactive management server node found 2025-06-29 03:49:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1fd06c56]) (logid:8f424ab7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:11,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3aa9e182]) (logid:b642359f) No inactive management server node found 2025-06-29 03:49:11,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3aa9e182]) (logid:b642359f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:12,308 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-337af744]) (logid:2210a0f4) HA health check task is running... 2025-06-29 03:49:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f0eab8e7]) (logid:5d559317) Found 0 routers to update status. 2025-06-29 03:49:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f0eab8e7]) (logid:5d559317) Found 0 VPC's to update Redundant State. 2025-06-29 03:49:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f0eab8e7]) (logid:5d559317) Found 0 networks to update RvR status. 2025-06-29 03:49:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65a029b]) (logid:a37ab2b2) No inactive management server node found 2025-06-29 03:49:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65a029b]) (logid:a37ab2b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f7a43da]) (logid:106535ab) No inactive management server node found 2025-06-29 03:49:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f7a43da]) (logid:106535ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:15,241 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8335: Seq 7-8335: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:15,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8335 2025-06-29 03:49:16,310 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6bb0acc5]) (logid:d72b2f00) HA health check task is running... 2025-06-29 03:49:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-228ebb98]) (logid:d3c7c75a) No inactive management server node found 2025-06-29 03:49:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-228ebb98]) (logid:d3c7c75a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-886fcc7f]) (logid:139dcaf4) No inactive management server node found 2025-06-29 03:49:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-886fcc7f]) (logid:139dcaf4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:19,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bdd79fa]) (logid:927ad61b) No inactive management server node found 2025-06-29 03:49:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bdd79fa]) (logid:927ad61b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:20,311 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b49bee53]) (logid:cdfdc926) HA health check task is running... 2025-06-29 03:49:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34cdfadd]) (logid:bb3cbfa9) No inactive management server node found 2025-06-29 03:49:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34cdfadd]) (logid:bb3cbfa9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:22,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fe4c6f2]) (logid:78fba6d0) No inactive management server node found 2025-06-29 03:49:22,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fe4c6f2]) (logid:78fba6d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5833f3cd]) (logid:cab63a00) No inactive management server node found 2025-06-29 03:49:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5833f3cd]) (logid:cab63a00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:24,313 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-15721d67]) (logid:c6e584e2) HA health check task is running... 2025-06-29 03:49:25,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8336: Seq 7-8336: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:25,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8336 2025-06-29 03:49:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-586c0751]) (logid:6eab54b9) No inactive management server node found 2025-06-29 03:49:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-586c0751]) (logid:6eab54b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffeeacf1]) (logid:08a53caf) No inactive management server node found 2025-06-29 03:49:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffeeacf1]) (logid:08a53caf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e4978fa9]) (logid:5cd832c4) Enabled non-edge zones available for scan: 1 2025-06-29 03:49:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e4978fa9]) (logid:5cd832c4) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:49:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e4978fa9]) (logid:5cd832c4) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:49:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-74cf2dfb]) (logid:a0071477) Enabled non-edge zones available for scan: 1 2025-06-29 03:49:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-74cf2dfb]) (logid:a0071477) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:49:28,315 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cfb16301]) (logid:7981b722) HA health check task is running... 2025-06-29 03:49:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7eab563]) (logid:e5ab83d6) No inactive management server node found 2025-06-29 03:49:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7eab563]) (logid:e5ab83d6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:49:29,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57089884]) (logid:c643a9c0) No inactive management server node found 2025-06-29 03:49:29,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57089884]) (logid:c643a9c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:31,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25afb1c4]) (logid:e27c45b8) No inactive management server node found 2025-06-29 03:49:31,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25afb1c4]) (logid:e27c45b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:32,317 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5d6de86a]) (logid:7681872a) HA health check task is running... 2025-06-29 03:49:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e20aacc]) (logid:993878f5) No inactive management server node found 2025-06-29 03:49:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e20aacc]) (logid:993878f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a24c15d]) (logid:314caca0) No inactive management server node found 2025-06-29 03:49:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a24c15d]) (logid:314caca0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:35,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8337: Seq 7-8337: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:35,252 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8337 2025-06-29 03:49:35,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f1bc803]) (logid:cdfe4997) No inactive management server node found 2025-06-29 03:49:35,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f1bc803]) (logid:cdfe4997) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:36,318 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-635a92c2]) (logid:9dba2a6c) HA health check task is running... 2025-06-29 03:49:37,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-884e5137]) (logid:5bc442d0) No inactive management server node found 2025-06-29 03:49:37,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-884e5137]) (logid:5bc442d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27094f5a]) (logid:800dbfce) No inactive management server node found 2025-06-29 03:49:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27094f5a]) (logid:800dbfce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:40,320 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9b1d3d25]) (logid:21823615) HA health check task is running... 2025-06-29 03:49:40,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fea5042a]) (logid:4e49045e) No inactive management server node found 2025-06-29 03:49:40,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fea5042a]) (logid:4e49045e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:41,875 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7daf3a2b]) (logid:818ac8c3) No inactive management server node found 2025-06-29 03:49:41,875 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7daf3a2b]) (logid:818ac8c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-7086ae47]) (logid:4183fe28) DbCollector is running... 2025-06-29 03:49:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d0270b40]) (logid:a0e8dd61) Found 0 routers to update status. 2025-06-29 03:49:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d0270b40]) (logid:a0e8dd61) Found 0 VPC's to update Redundant State. 2025-06-29 03:49:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d0270b40]) (logid:a0e8dd61) Found 0 networks to update RvR status. 2025-06-29 03:49:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f3592a3]) (logid:6cfa8df8) No inactive management server node found 2025-06-29 03:49:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f3592a3]) (logid:6cfa8df8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:49:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) ManagementServerCollector is running... 2025-06-29 03:49:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) DB properties were already loaded 2025-06-29 03:49:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:49:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Execution of process [4065371] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:49:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Exit value of process [4065371] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:49:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Process [4065371] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:49:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) usage install: null 2025-06-29 03:49:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Sessions active: 0 2025-06-29 03:49:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:49:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065374] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:49:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:49:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065379] for command [/bin/bash -c uname -r ]. 2025-06-29 03:49:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:49:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065380] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:49:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) system memory from /proc: 4887515136 2025-06-29 03:49:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:49:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065385] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:49:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) free memory from /proc: 188354560 2025-06-29 03:49:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:49:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065390] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:49:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) used memory from /proc: 1071852 2025-06-29 03:49:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:49:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065391] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:49:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:49:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065392] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:49:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:49:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065393] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:49:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:49:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065400] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:49:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:49:44,131 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Successfully executed process [4065401] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:49:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail buffersdirect.capacity, 242407 2025-06-29 03:49:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail buffersdirect.count, 74 2025-06-29 03:49:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail buffersdirect.used, 242407 2025-06-29 03:49:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail buffersmapped.capacity, 0 2025-06-29 03:49:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail buffersmapped.count, 0 2025-06-29 03:49:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail buffersmapped.used, 0 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail jvmuptime, 843837 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memoryheap.committed, 545259520 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memoryheap.init, 77594624 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memoryheap.usage, 0.19959390415525868 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorynon-heap.committed, 190316544 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorynon-heap.max, -1 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorynon-heap.usage, -1.88238528E8 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorynon-heap.used, 188238528 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09622017462010131 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20821287532908989 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.Compressed-Class-Space.usage, 0.01219707727432251 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.Metaspace.usage, 0.9939946319477719 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.PS-Eden-Space.usage, 0.2917855334301577 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:49:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:49:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorytotal.committed, 735576064 2025-06-29 03:49:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorytotal.init, 85262336 2025-06-29 03:49:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorytotal.max, 1908932607 2025-06-29 03:49:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail memorytotal.used, 569252272 2025-06-29 03:49:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail threadsdeadlocks, [] 2025-06-29 03:49:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail threadsnew.count, 0 2025-06-29 03:49:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:49:44,139 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-8a4c246d]) (logid:b0736009) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":165265216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":843760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":568222080,"totalCommitted":735576064,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381011312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":188354560,"systemMemoryUsed":1071852,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.15,0.17],"systemCyclesUsage":[66399612,53823797,4588942996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:49:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 27, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":165265216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":843760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":568222080,"totalCommitted":735576064,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381011312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":188354560,"systemMemoryUsed":1071852,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.15,0.17],"systemCyclesUsage":[66399612,53823797,4588942996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:49:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:49:44,148 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-10:[ctx-1e9e56b3]) (logid:8be88ba1) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":165265216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":843760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":568222080,"totalCommitted":735576064,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381011312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":188354560,"systemMemoryUsed":1071852,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.15,0.17],"systemCyclesUsage":[66399612,53823797,4588942996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:49:44,148 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:49:44,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 27, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":165265216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":843760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":568222080,"totalCommitted":735576064,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381011312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":188354560,"systemMemoryUsed":1071852,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.15,0.17],"systemCyclesUsage":[66399612,53823797,4588942996],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:49:44,321 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2b10300d]) (logid:8b63d4d4) HA health check task is running... 2025-06-29 03:49:44,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-afdede9d]) (logid:61c802de) No inactive management server node found 2025-06-29 03:49:44,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-afdede9d]) (logid:61c802de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:45,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8338: Seq 7-8338: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:45,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8338 2025-06-29 03:49:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:49:45,926 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:49:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a14583ec]) (logid:8dbb9a92) No inactive management server node found 2025-06-29 03:49:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a14583ec]) (logid:8dbb9a92) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:49:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:49:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:49:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:49:47,094 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:49:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:49:47,097 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:49:47,100 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:49:47,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20c2ab3a]) (logid:87569a26) No inactive management server node found 2025-06-29 03:49:47,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20c2ab3a]) (logid:87569a26) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-05aa787a]) (logid:6364ae03) ClusterDRS.poll is being called at 2025-06-29T01:50:00+0000 2025-06-29 03:49:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-05aa787a]) (logid:6364ae03) Removed 0 old drs migration plans 2025-06-29 03:49:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-da81f3b4]) (logid:285e3e3d) VM scheduler.poll is being called at 2025-06-29T01:50:00+0000 2025-06-29 03:49:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-da81f3b4]) (logid:285e3e3d) Got 0 scheduled jobs to be executed at 2025-06-29T01:50:00+0000 2025-06-29 03:49:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-da81f3b4]) (logid:285e3e3d) Cleaned up 0 VM scheduled job entries 2025-06-29 03:49:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-51eaf031]) (logid:98e09c06) VM Operation Thread Running 2025-06-29 03:49:48,324 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-27a5264c]) (logid:083f814a) HA health check task is running... 2025-06-29 03:49:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-842ce2d8]) (logid:3c3b0be3) No inactive management server node found 2025-06-29 03:49:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-842ce2d8]) (logid:3c3b0be3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:50,241 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8340: Seq 7-8340: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:49:50,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8340 2025-06-29 03:49:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28631f82]) (logid:bb204432) No inactive management server node found 2025-06-29 03:49:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28631f82]) (logid:bb204432) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:52,325 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2f67c686]) (logid:95b1bc6f) HA health check task is running... 2025-06-29 03:49:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad12dd0e]) (logid:cb927bd5) No inactive management server node found 2025-06-29 03:49:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad12dd0e]) (logid:cb927bd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9304206f]) (logid:646a5388) No inactive management server node found 2025-06-29 03:49:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9304206f]) (logid:646a5388) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:55,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec15dcd1]) (logid:8c411b92) No inactive management server node found 2025-06-29 03:49:55,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec15dcd1]) (logid:8c411b92) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:56,326 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4d7021c8]) (logid:aef58b87) HA health check task is running... 2025-06-29 03:49:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-448e2dc9]) (logid:e01c29aa) No inactive management server node found 2025-06-29 03:49:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-448e2dc9]) (logid:e01c29aa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:58,081 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-5a1983b2]) (logid:1e067eab) AutoScaling Monitor is running... 2025-06-29 03:49:58,084 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-52887aee]) (logid:baa5a35a) Starting to remove all volume_stats rows older than [Sat Jun 28 15:49:58 CEST 2025]. 2025-06-29 03:49:58,085 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-52887aee]) (logid:baa5a35a) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:49:58 CEST 2025]. 2025-06-29 03:49:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bf1e5c37]) (logid:38e88672) Enabled non-edge zones available for scan: 1 2025-06-29 03:49:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bf1e5c37]) (logid:38e88672) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:49:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bf1e5c37]) (logid:38e88672) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:49:58,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c447fbf7]) (logid:cf7bec58) Enabled non-edge zones available for scan: 1 2025-06-29 03:49:58,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c447fbf7]) (logid:cf7bec58) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:49:58,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-163bde62]) (logid:cbb90b55) No inactive management server node found 2025-06-29 03:49:58,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-163bde62]) (logid:cbb90b55) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:49:58,458 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-2498277b]) (logid:5d5266f3) Starting to remove all vm_stats rows older than [Sat Jun 28 15:49:58 CEST 2025]. 2025-06-29 03:49:58,481 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-2498277b]) (logid:5d5266f3) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:49:58 CEST 2025]. 2025-06-29 03:49:59,034 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:49:59,036 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:49:59,038 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) Seq 2-4725683383995203647: Routed from 90520738888109 2025-06-29 03:49:59,082 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) Seq 1-4725683383995203647: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:49:59,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:49:59,084 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) Seq 2-3926857400090361927: Routed from 90520738888109 2025-06-29 03:49:59,129 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-15668088]) (logid:8a19b3aa) Seq 2-3926857400090361927: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:49:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e60f7b95]) (logid:1b6b8ddc) No inactive management server node found 2025-06-29 03:49:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e60f7b95]) (logid:1b6b8ddc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:00,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8341: Seq 7-8341: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:00,252 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8341 2025-06-29 03:50:00,328 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2324cef7]) (logid:a63fe4ca) HA health check task is running... 2025-06-29 03:50:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dcf82e5]) (logid:847381ed) No inactive management server node found 2025-06-29 03:50:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dcf82e5]) (logid:847381ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:01,682 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:50:01,686 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:50:01,687 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) Seq 2-4725683383995203648: Routed from 90520738888109 2025-06-29 03:50:01,745 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) Seq 1-4725683383995203648: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:50:01,784 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:50:01,785 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) Seq 2-3926857400090361928: Routed from 90520738888109 2025-06-29 03:50:01,881 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-81161e7d]) (logid:3c82c872) Seq 2-3926857400090361928: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:50:02,124 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) StorageCollector is running... 2025-06-29 03:50:02,132 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:50:02,132 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:50:02,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:50:02,133 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-3938116399158788117: Routed from 90520738888109 2025-06-29 03:50:02,187 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 4-3938116399158788117: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:50:02,191 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:50:02,191 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:50:02,191 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:50:02,193 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-3926857400090361929: Routed from 90520738888109 2025-06-29 03:50:02,204 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-3926857400090361929: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:50:02,227 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:50:02,227 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:50:02,227 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:50:02,228 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-4725683383995203649: Routed from 90520738888109 2025-06-29 03:50:02,276 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 1-4725683383995203649: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:50:02,290 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:50:02,290 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:50:02,290 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:50:02,291 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-4725683383995203650: Routed from 90520738888109 2025-06-29 03:50:02,378 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 1-4725683383995203650: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:50:02,380 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:50:02,380 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:50:02,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:50:02,381 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-3926857400090361930: Routed from 90520738888109 2025-06-29 03:50:02,425 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-72f12f8f]) (logid:ad7b50b4) Seq 2-3926857400090361930: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:50:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d30b323]) (logid:809fd959) No inactive management server node found 2025-06-29 03:50:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d30b323]) (logid:809fd959) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:04,329 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-314f3771]) (logid:8718d5c2) HA health check task is running... 2025-06-29 03:50:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7174f3ac]) (logid:da95ba64) No inactive management server node found 2025-06-29 03:50:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7174f3ac]) (logid:da95ba64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72e171f6]) (logid:97e14f1c) No inactive management server node found 2025-06-29 03:50:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72e171f6]) (logid:97e14f1c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6793863a]) (logid:94d06241) No inactive management server node found 2025-06-29 03:50:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6793863a]) (logid:94d06241) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:08,331 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7814d2b6]) (logid:3e6d403c) HA health check task is running... 2025-06-29 03:50:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f67b2df]) (logid:f18f6b29) No inactive management server node found 2025-06-29 03:50:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f67b2df]) (logid:f18f6b29) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:10,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8342: Seq 7-8342: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:10,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8342 2025-06-29 03:50:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d2c87aa]) (logid:ba783cbf) No inactive management server node found 2025-06-29 03:50:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d2c87aa]) (logid:ba783cbf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:11,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb57113]) (logid:38cbe674) No inactive management server node found 2025-06-29 03:50:11,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cb57113]) (logid:38cbe674) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:12,332 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d116bf1e]) (logid:06f3add3) HA health check task is running... 2025-06-29 03:50:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-46d34843]) (logid:d77d1893) Found 0 routers to update status. 2025-06-29 03:50:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-46d34843]) (logid:d77d1893) Found 0 VPC's to update Redundant State. 2025-06-29 03:50:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-46d34843]) (logid:d77d1893) Found 0 networks to update RvR status. 2025-06-29 03:50:13,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-865fa5ad]) (logid:323d1a2b) No inactive management server node found 2025-06-29 03:50:13,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-865fa5ad]) (logid:323d1a2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f9d7898]) (logid:093bb4b6) No inactive management server node found 2025-06-29 03:50:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f9d7898]) (logid:093bb4b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:15,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8343: Seq 7-8343: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:15,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8343 2025-06-29 03:50:16,334 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d195d863]) (logid:5df9281d) HA health check task is running... 2025-06-29 03:50:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-614243fd]) (logid:5f99e99f) No inactive management server node found 2025-06-29 03:50:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-614243fd]) (logid:5f99e99f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6022d8e3]) (logid:81e621a9) No inactive management server node found 2025-06-29 03:50:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6022d8e3]) (logid:81e621a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7c1289dc]) (logid:9334606f) Resetting hosts suitable for reconnect 2025-06-29 03:50:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7c1289dc]) (logid:9334606f) Completed resetting hosts suitable for reconnect 2025-06-29 03:50:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7c1289dc]) (logid:9334606f) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:50:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7c1289dc]) (logid:9334606f) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:50:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7c1289dc]) (logid:9334606f) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:50:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7c1289dc]) (logid:9334606f) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:50:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bdc636a]) (logid:93331dcb) No inactive management server node found 2025-06-29 03:50:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bdc636a]) (logid:93331dcb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:20,336 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6425bb72]) (logid:c478d34a) HA health check task is running... 2025-06-29 03:50:20,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24ae3b37]) (logid:625ac7cb) No inactive management server node found 2025-06-29 03:50:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24ae3b37]) (logid:625ac7cb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:50:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41ed2f32]) (logid:d091ab36) No inactive management server node found 2025-06-29 03:50:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41ed2f32]) (logid:d091ab36) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96425788]) (logid:56cd48b8) No inactive management server node found 2025-06-29 03:50:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96425788]) (logid:56cd48b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:24,338 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-eeed3a95]) (logid:fdf1e012) HA health check task is running... 2025-06-29 03:50:25,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8344: Seq 7-8344: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:25,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8344 2025-06-29 03:50:25,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e2d91b1]) (logid:fa96c2f9) No inactive management server node found 2025-06-29 03:50:25,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e2d91b1]) (logid:fa96c2f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4980039]) (logid:ae8d3c94) No inactive management server node found 2025-06-29 03:50:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4980039]) (logid:ae8d3c94) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9ed346d6]) (logid:593bf8b8) Enabled non-edge zones available for scan: 1 2025-06-29 03:50:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9ed346d6]) (logid:593bf8b8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:50:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9ed346d6]) (logid:593bf8b8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:50:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0366b590]) (logid:9a07b3aa) Enabled non-edge zones available for scan: 1 2025-06-29 03:50:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0366b590]) (logid:9a07b3aa) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:50:28,339 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1239a2a4]) (logid:b057b34f) HA health check task is running... 2025-06-29 03:50:28,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-560aa9ab]) (logid:a4299cd4) No inactive management server node found 2025-06-29 03:50:28,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-560aa9ab]) (logid:a4299cd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b47e57a]) (logid:4fa84957) No inactive management server node found 2025-06-29 03:50:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b47e57a]) (logid:4fa84957) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d335ae7b]) (logid:6c6028bf) No inactive management server node found 2025-06-29 03:50:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d335ae7b]) (logid:6c6028bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:32,340 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-318046ee]) (logid:cbc33ccf) HA health check task is running... 2025-06-29 03:50:32,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7f1b950]) (logid:c87221fc) No inactive management server node found 2025-06-29 03:50:32,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7f1b950]) (logid:c87221fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c25f5885]) (logid:4325bf58) No inactive management server node found 2025-06-29 03:50:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c25f5885]) (logid:4325bf58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:35,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8345: Seq 7-8345: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:35,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8345 2025-06-29 03:50:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f258ec48]) (logid:e7102513) No inactive management server node found 2025-06-29 03:50:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f258ec48]) (logid:e7102513) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:36,342 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6328739a]) (logid:76043722) HA health check task is running... 2025-06-29 03:50:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bf79dd3]) (logid:eb9089b7) No inactive management server node found 2025-06-29 03:50:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bf79dd3]) (logid:eb9089b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb04ea15]) (logid:5d0e5c80) No inactive management server node found 2025-06-29 03:50:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb04ea15]) (logid:5d0e5c80) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:40,343 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5c51e811]) (logid:aacc9c41) HA health check task is running... 2025-06-29 03:50:40,369 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f628771d]) (logid:ff53f263) No inactive management server node found 2025-06-29 03:50:40,369 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f628771d]) (logid:ff53f263) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1961cdd8]) (logid:70ecfde4) No inactive management server node found 2025-06-29 03:50:41,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1961cdd8]) (logid:70ecfde4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-a9a6ee69]) (logid:69d273a7) DbCollector is running... 2025-06-29 03:50:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8ed8a0b]) (logid:b57743f9) Found 0 routers to update status. 2025-06-29 03:50:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8ed8a0b]) (logid:b57743f9) Found 0 VPC's to update Redundant State. 2025-06-29 03:50:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8ed8a0b]) (logid:b57743f9) Found 0 networks to update RvR status. 2025-06-29 03:50:43,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a033290c]) (logid:129ca993) No inactive management server node found 2025-06-29 03:50:43,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a033290c]) (logid:129ca993) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) ManagementServerCollector is running... 2025-06-29 03:50:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) DB properties were already loaded 2025-06-29 03:50:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:50:44,055 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Execution of process [4065414] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:50:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Exit value of process [4065414] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:50:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Process [4065414] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:50:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) usage install: null 2025-06-29 03:50:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Sessions active: 0 2025-06-29 03:50:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:50:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065417] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:50:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:50:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065422] for command [/bin/bash -c uname -r ]. 2025-06-29 03:50:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:50:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065423] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:50:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) system memory from /proc: 4887515136 2025-06-29 03:50:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:50:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065428] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:50:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) free memory from /proc: 180514816 2025-06-29 03:50:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:50:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065433] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:50:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) used memory from /proc: 1072492 2025-06-29 03:50:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:50:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065434] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:50:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:50:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065435] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:50:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:50:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065436] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:50:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:50:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065443] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:50:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:50:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Successfully executed process [4065444] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail buffersdirect.capacity, 242407 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail buffersdirect.count, 74 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail buffersdirect.used, 242407 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail buffersmapped.capacity, 0 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail buffersmapped.count, 0 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail buffersmapped.used, 0 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail jvmuptime, 903831 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memoryheap.committed, 545259520 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memoryheap.init, 77594624 2025-06-29 03:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memoryheap.usage, 0.21172332135048322 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorynon-heap.committed, 190644224 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorynon-heap.max, -1 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorynon-heap.usage, -1.88661632E8 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorynon-heap.used, 188661632 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09692519494801387 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20994163861765588 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.Compressed-Class-Space.usage, 0.012198105454444885 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.Metaspace.usage, 0.9944164524589919 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.PS-Eden-Space.usage, 0.3281937840549 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorytotal.committed, 735903744 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorytotal.init, 85262336 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorytotal.max, 1908932607 2025-06-29 03:50:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail memorytotal.used, 592827184 2025-06-29 03:50:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail threadsdeadlocks, [] 2025-06-29 03:50:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail threadsnew.count, 0 2025-06-29 03:50:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:50:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-d06890d8]) (logid:ba84f5c6) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":142267048,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":903759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":591641184,"totalCommitted":735903744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":404165552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":180514816,"systemMemoryUsed":1072492,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.12,0.16],"systemCyclesUsage":[66399954,53824103,4588966105],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:50:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 29, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":142267048,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":903759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":591641184,"totalCommitted":735903744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":404165552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":180514816,"systemMemoryUsed":1072492,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.12,0.16],"systemCyclesUsage":[66399954,53824103,4588966105],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:50:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:50:44,141 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-10:[ctx-31009386]) (logid:317861b3) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":142267048,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":903759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":591641184,"totalCommitted":735903744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":404165552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":180514816,"systemMemoryUsed":1072492,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.12,0.16],"systemCyclesUsage":[66399954,53824103,4588966105],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:50:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 03:50:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 29, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":142267048,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":903759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":591641184,"totalCommitted":735903744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":404165552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":104,"systemMemoryTotal":4887515136,"systemMemoryFree":180514816,"systemMemoryUsed":1072492,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.12,0.16],"systemCyclesUsage":[66399954,53824103,4588966105],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:50:44,345 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f1b2e122]) (logid:7c7c5789) HA health check task is running... 2025-06-29 03:50:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bffb6cd6]) (logid:672fd7c8) No inactive management server node found 2025-06-29 03:50:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bffb6cd6]) (logid:672fd7c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:45,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8346: Seq 7-8346: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:45,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8346 2025-06-29 03:50:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:50:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:50:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ce7c05e]) (logid:c3025704) No inactive management server node found 2025-06-29 03:50:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ce7c05e]) (logid:c3025704) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:46,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:50:46,652 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:50:46,654 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:50:46,656 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:50:47,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:50:47,094 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:50:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:50:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:50:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6248f87f]) (logid:067c0fac) No inactive management server node found 2025-06-29 03:50:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6248f87f]) (logid:067c0fac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1426f2c5]) (logid:8d434b00) ClusterDRS.poll is being called at 2025-06-29T01:51:00+0000 2025-06-29 03:50:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1426f2c5]) (logid:8d434b00) Removed 0 old drs migration plans 2025-06-29 03:50:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7bee6fd5]) (logid:45292345) VM scheduler.poll is being called at 2025-06-29T01:51:00+0000 2025-06-29 03:50:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7bee6fd5]) (logid:45292345) Got 0 scheduled jobs to be executed at 2025-06-29T01:51:00+0000 2025-06-29 03:50:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7bee6fd5]) (logid:45292345) Cleaned up 0 VM scheduled job entries 2025-06-29 03:50:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9702d046]) (logid:4929a9f1) VM Operation Thread Running 2025-06-29 03:50:48,346 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b800e566]) (logid:06e1d24e) HA health check task is running... 2025-06-29 03:50:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66789d93]) (logid:6d51afa7) No inactive management server node found 2025-06-29 03:50:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66789d93]) (logid:6d51afa7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72179520]) (logid:4a87131b) No inactive management server node found 2025-06-29 03:50:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72179520]) (logid:4a87131b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:52,347 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ef9576aa]) (logid:47ef594d) HA health check task is running... 2025-06-29 03:50:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c90dc01a]) (logid:18d0a718) No inactive management server node found 2025-06-29 03:50:52,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c90dc01a]) (logid:18d0a718) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dc90240]) (logid:cb4ce91a) No inactive management server node found 2025-06-29 03:50:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dc90240]) (logid:cb4ce91a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:55,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8348: Seq 7-8348: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:50:55,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8348 2025-06-29 03:50:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b5a376]) (logid:06a7b28d) No inactive management server node found 2025-06-29 03:50:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b5a376]) (logid:06a7b28d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:56,349 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d3790b21]) (logid:f6fcef8c) HA health check task is running... 2025-06-29 03:50:56,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19c15c10]) (logid:e6f5fb30) No inactive management server node found 2025-06-29 03:50:56,875 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19c15c10]) (logid:e6f5fb30) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:58,083 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-321dd855]) (logid:82be9cab) AutoScaling Monitor is running... 2025-06-29 03:50:58,087 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-7df81ab4]) (logid:3f3ccf9c) Starting to remove all volume_stats rows older than [Sat Jun 28 15:50:58 CEST 2025]. 2025-06-29 03:50:58,088 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-7df81ab4]) (logid:3f3ccf9c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:50:58 CEST 2025]. 2025-06-29 03:50:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bf9d1b7b]) (logid:d23cc753) Enabled non-edge zones available for scan: 1 2025-06-29 03:50:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bf9d1b7b]) (logid:d23cc753) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:50:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bf9d1b7b]) (logid:d23cc753) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:50:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fffd5c78]) (logid:0380e5cf) Enabled non-edge zones available for scan: 1 2025-06-29 03:50:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fffd5c78]) (logid:0380e5cf) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:50:58,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbd30420]) (logid:83e278d3) No inactive management server node found 2025-06-29 03:50:58,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbd30420]) (logid:83e278d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:50:58,483 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-2acce083]) (logid:5e0372a4) Starting to remove all vm_stats rows older than [Sat Jun 28 15:50:58 CEST 2025]. 2025-06-29 03:50:58,515 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-2acce083]) (logid:5e0372a4) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:50:58 CEST 2025]. 2025-06-29 03:50:59,132 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:50:59,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:50:59,136 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) Seq 2-4725683383995203651: Routed from 90520738888109 2025-06-29 03:50:59,180 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) Seq 1-4725683383995203651: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:50:59,181 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:50:59,182 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) Seq 2-3926857400090361931: Routed from 90520738888109 2025-06-29 03:50:59,227 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-70655bbb]) (logid:5f5ff675) Seq 2-3926857400090361931: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:50:59,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97501f79]) (logid:ff46bb84) No inactive management server node found 2025-06-29 03:50:59,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97501f79]) (logid:ff46bb84) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:00,350 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b580883e]) (logid:e97c19a7) HA health check task is running... 2025-06-29 03:51:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6809349c]) (logid:7ce4c12d) No inactive management server node found 2025-06-29 03:51:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6809349c]) (logid:7ce4c12d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:01,939 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:51:01,942 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:51:01,943 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) Seq 2-4725683383995203652: Routed from 90520738888109 2025-06-29 03:51:01,997 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) Seq 1-4725683383995203652: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:51:02,047 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:51:02,048 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) Seq 2-3926857400090361932: Routed from 90520738888109 2025-06-29 03:51:02,198 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-519b95f4]) (logid:fb5fc491) Seq 2-3926857400090361932: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:51:02,426 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) StorageCollector is running... 2025-06-29 03:51:02,432 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:51:02,432 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:51:02,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:51:02,434 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3938116399158788118: Routed from 90520738888109 2025-06-29 03:51:02,490 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 4-3938116399158788118: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:51:02,495 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:51:02,495 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:51:02,495 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:51:02,497 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3926857400090361933: Routed from 90520738888109 2025-06-29 03:51:02,510 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3926857400090361933: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:51:02,521 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:51:02,521 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:51:02,521 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:51:02,523 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3926857400090361934: Routed from 90520738888109 2025-06-29 03:51:02,617 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3926857400090361934: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:51:02,627 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:51:02,627 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:51:02,627 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:51:02,628 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-4725683383995203653: Routed from 90520738888109 2025-06-29 03:51:02,674 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 1-4725683383995203653: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:51:02,676 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:51:02,676 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:51:02,676 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:51:02,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3926857400090361935: Routed from 90520738888109 2025-06-29 03:51:02,723 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d565f792]) (logid:8d487691) Seq 2-3926857400090361935: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:51:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3dfa7215]) (logid:7a267cb9) No inactive management server node found 2025-06-29 03:51:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3dfa7215]) (logid:7a267cb9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:04,352 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9b9bcd2f]) (logid:b9ebb202) HA health check task is running... 2025-06-29 03:51:04,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16d30454]) (logid:60bfb77b) No inactive management server node found 2025-06-29 03:51:04,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16d30454]) (logid:60bfb77b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:05,244 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8349: Seq 7-8349: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:51:05,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8349 2025-06-29 03:51:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-602cc769]) (logid:897a1def) No inactive management server node found 2025-06-29 03:51:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-602cc769]) (logid:897a1def) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0aae5ead]) (logid:ac8652b9) No inactive management server node found 2025-06-29 03:51:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0aae5ead]) (logid:ac8652b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:08,353 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8b7205ae]) (logid:725ab420) HA health check task is running... 2025-06-29 03:51:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d7f67e7]) (logid:6af8d588) No inactive management server node found 2025-06-29 03:51:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d7f67e7]) (logid:6af8d588) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eba4b592]) (logid:3ea91fa4) No inactive management server node found 2025-06-29 03:51:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eba4b592]) (logid:3ea91fa4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48277d23]) (logid:4a53c706) No inactive management server node found 2025-06-29 03:51:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48277d23]) (logid:4a53c706) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:12,354 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c984e679]) (logid:4025d21d) HA health check task is running... 2025-06-29 03:51:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5d9c108b]) (logid:1f3904f5) Found 0 routers to update status. 2025-06-29 03:51:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5d9c108b]) (logid:1f3904f5) Found 0 VPC's to update Redundant State. 2025-06-29 03:51:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5d9c108b]) (logid:1f3904f5) Found 0 networks to update RvR status. 2025-06-29 03:51:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4539c90f]) (logid:6b18ee67) No inactive management server node found 2025-06-29 03:51:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4539c90f]) (logid:6b18ee67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45a40bce]) (logid:54203aa4) No inactive management server node found 2025-06-29 03:51:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45a40bce]) (logid:54203aa4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:15,248 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8350: Seq 7-8350: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:51:15,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8350 2025-06-29 03:51:16,356 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4ddda2e0]) (logid:d04a98e6) HA health check task is running... 2025-06-29 03:51:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0381347c]) (logid:a40f439c) No inactive management server node found 2025-06-29 03:51:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0381347c]) (logid:a40f439c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:17,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50aebda0]) (logid:caf03279) No inactive management server node found 2025-06-29 03:51:17,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50aebda0]) (logid:caf03279) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e1f8c3a]) (logid:f11c3263) No inactive management server node found 2025-06-29 03:51:19,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e1f8c3a]) (logid:f11c3263) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:20,357 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-82456ad0]) (logid:2d99ffee) HA health check task is running... 2025-06-29 03:51:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5911df58]) (logid:f1681e36) No inactive management server node found 2025-06-29 03:51:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5911df58]) (logid:f1681e36) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-463d54f7]) (logid:5bbeeee0) No inactive management server node found 2025-06-29 03:51:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-463d54f7]) (logid:5bbeeee0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cfa6129]) (logid:2c7ca474) No inactive management server node found 2025-06-29 03:51:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cfa6129]) (logid:2c7ca474) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:24,359 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5bda9013]) (logid:d758b4fe) HA health check task is running... 2025-06-29 03:51:25,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8351: Seq 7-8351: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:51:25,267 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8351 2025-06-29 03:51:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6b2aabd]) (logid:a3664f60) No inactive management server node found 2025-06-29 03:51:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6b2aabd]) (logid:a3664f60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb601c5f]) (logid:cb6f9c68) No inactive management server node found 2025-06-29 03:51:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb601c5f]) (logid:cb6f9c68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61fab329]) (logid:9d2b9e14) Enabled non-edge zones available for scan: 1 2025-06-29 03:51:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-61fab329]) (logid:9d2b9e14) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:51:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-61fab329]) (logid:9d2b9e14) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:51:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dc0a303c]) (logid:94c8177c) Enabled non-edge zones available for scan: 1 2025-06-29 03:51:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dc0a303c]) (logid:94c8177c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:51:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e34cfeb2]) (logid:062a318c) No inactive management server node found 2025-06-29 03:51:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e34cfeb2]) (logid:062a318c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:28,360 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3bac2b37]) (logid:2640cd86) HA health check task is running... 2025-06-29 03:51:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d831df21]) (logid:4d595712) No inactive management server node found 2025-06-29 03:51:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d831df21]) (logid:4d595712) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:31,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf6109e1]) (logid:aa9d7920) No inactive management server node found 2025-06-29 03:51:31,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf6109e1]) (logid:aa9d7920) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:32,361 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-abc1765a]) (logid:c47e8f81) HA health check task is running... 2025-06-29 03:51:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-534fd7a9]) (logid:57a64626) No inactive management server node found 2025-06-29 03:51:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-534fd7a9]) (logid:57a64626) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ecdbca9]) (logid:05c63f80) No inactive management server node found 2025-06-29 03:51:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ecdbca9]) (logid:05c63f80) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:35,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8352: Seq 7-8352: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:51:35,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8352 2025-06-29 03:51:35,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-497e574a]) (logid:7da85ffe) No inactive management server node found 2025-06-29 03:51:35,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-497e574a]) (logid:7da85ffe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:36,363 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8b9a1bcf]) (logid:546de5e9) HA health check task is running... 2025-06-29 03:51:37,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1b81ca1]) (logid:4b5c0441) No inactive management server node found 2025-06-29 03:51:37,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1b81ca1]) (logid:4b5c0441) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:38,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35e0d678]) (logid:b6411fa9) No inactive management server node found 2025-06-29 03:51:38,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35e0d678]) (logid:b6411fa9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12d92d6f]) (logid:e6bcb146) No inactive management server node found 2025-06-29 03:51:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12d92d6f]) (logid:e6bcb146) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:40,364 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7f45ae6c]) (logid:12d3b529) HA health check task is running... 2025-06-29 03:51:41,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2703c6ac]) (logid:702acc05) No inactive management server node found 2025-06-29 03:51:41,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2703c6ac]) (logid:702acc05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-008ec84e]) (logid:afa63c12) DbCollector is running... 2025-06-29 03:51:43,093 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-cb5b5f18]) (logid:f6206952) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:51:43,094 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-cb5b5f18]) (logid:f6206952) Snapshot scheduler is being called at [2025-06-29T01:51:43+0000]. 2025-06-29 03:51:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-cb5b5f18]) (logid:f6206952) There are [0] scheduled snapshots to be executed at [2025-06-29T01:51:43+0000]. 2025-06-29 03:51:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-7bc12ee0]) (logid:6bdf43e3) Found 0 running routers. 2025-06-29 03:51:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8feb685b]) (logid:406e3793) Found 0 routers to update status. 2025-06-29 03:51:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8feb685b]) (logid:406e3793) Found 0 VPC's to update Redundant State. 2025-06-29 03:51:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8feb685b]) (logid:406e3793) Found 0 networks to update RvR status. 2025-06-29 03:51:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-d31f44ae]) (logid:24e33d0d) External devices are not used. Skipping external device usage collection 2025-06-29 03:51:43,125 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-cb27d62b]) (logid:49c3abde) Started resource counters recalculation periodic task. 2025-06-29 03:51:43,127 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-cb27d62b]) (logid:49c3abde) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:51:43,304 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-70ebc9b6]) (logid:b1334e85) Backup backup.poll is being called at 2025-06-29T01:51:43+0000 2025-06-29 03:51:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-617c90a2]) (logid:4fe1ad19) No inactive management server node found 2025-06-29 03:51:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-617c90a2]) (logid:4fe1ad19) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:43,535 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-cb27d62b]) (logid:49c3abde) Finished resource counters recalculation periodic task. 2025-06-29 03:51:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) ManagementServerCollector is running... 2025-06-29 03:51:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) DB properties were already loaded 2025-06-29 03:51:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:51:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Execution of process [4065450] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:51:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Exit value of process [4065450] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:51:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Process [4065450] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:51:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) usage install: null 2025-06-29 03:51:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Sessions active: 0 2025-06-29 03:51:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:51:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065453] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:51:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:51:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065458] for command [/bin/bash -c uname -r ]. 2025-06-29 03:51:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:51:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065459] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:51:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) system memory from /proc: 4887515136 2025-06-29 03:51:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:51:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065464] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:51:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) free memory from /proc: 185237504 2025-06-29 03:51:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:51:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065469] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:51:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) used memory from /proc: 1072748 2025-06-29 03:51:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:51:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065470] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:51:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:51:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065471] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:51:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:51:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065472] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:51:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:51:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065479] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:51:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:51:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Successfully executed process [4065480] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:51:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail buffersdirect.capacity, 242508 2025-06-29 03:51:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail buffersdirect.count, 75 2025-06-29 03:51:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail buffersdirect.used, 242508 2025-06-29 03:51:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail buffersmapped.capacity, 0 2025-06-29 03:51:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail buffersmapped.count, 0 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail buffersmapped.used, 0 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail jvmuptime, 963834 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memoryheap.committed, 545259520 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memoryheap.init, 77594624 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memoryheap.usage, 0.22895854058353432 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorynon-heap.committed, 190971904 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorynon-heap.max, -1 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorynon-heap.usage, -1.86014632E8 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorynon-heap.used, 186014632 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09507048120501199 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.18987444596260872 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.Compressed-Class-Space.usage, 0.012200161814689636 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.Metaspace.usage, 0.993794798052482 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.PS-Eden-Space.usage, 0.3799278592630938 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorytotal.committed, 736231424 2025-06-29 03:51:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorytotal.init, 85262336 2025-06-29 03:51:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorytotal.max, 1908932607 2025-06-29 03:51:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail memorytotal.used, 623081056 2025-06-29 03:51:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail threadsdeadlocks, [] 2025-06-29 03:51:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail threadsnew.count, 0 2025-06-29 03:51:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:51:44,135 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-a8173626]) (logid:dcc61671) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":109111120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":963760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":622153784,"totalCommitted":736231424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":437066424,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":185237504,"systemMemoryUsed":1072748,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.16],"systemCyclesUsage":[66400339,53824415,4588989164],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:51:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 31, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":109111120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":963760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":622153784,"totalCommitted":736231424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":437066424,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":185237504,"systemMemoryUsed":1072748,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.16],"systemCyclesUsage":[66400339,53824415,4588989164],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:51:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:51:44,145 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 03:51:44,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 31, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":109111120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":963760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":622153784,"totalCommitted":736231424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":437066424,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":185237504,"systemMemoryUsed":1072748,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.16],"systemCyclesUsage":[66400339,53824415,4588989164],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:51:44,145 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-11:[ctx-65ae427c]) (logid:19b294b5) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":109111120,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":963760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":622153784,"totalCommitted":736231424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":437066424,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":185237504,"systemMemoryUsed":1072748,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.11,0.16],"systemCyclesUsage":[66400339,53824415,4588989164],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:51:44,365 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-835fb582]) (logid:d5794858) HA health check task is running... 2025-06-29 03:51:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-015c7079]) (logid:a6e39ae1) No inactive management server node found 2025-06-29 03:51:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-015c7079]) (logid:a6e39ae1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:45,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8353: Seq 7-8353: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:51:45,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8353 2025-06-29 03:51:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:51:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:51:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e92ad20]) (logid:53fc3c98) No inactive management server node found 2025-06-29 03:51:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e92ad20]) (logid:53fc3c98) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:46,646 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:51:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:51:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:51:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:51:47,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:51:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:51:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:51:47,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:51:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-885f70ce]) (logid:faef3d95) No inactive management server node found 2025-06-29 03:51:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-885f70ce]) (logid:faef3d95) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:48,039 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-2:[ctx-c00c79c5]) (logid:006d8df2) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:51:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2f53cb3f]) (logid:a5fe1e2d) ClusterDRS.poll is being called at 2025-06-29T01:52:00+0000 2025-06-29 03:51:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2f53cb3f]) (logid:a5fe1e2d) Removed 0 old drs migration plans 2025-06-29 03:51:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f2fe293b]) (logid:232bdc06) Resetting hosts suitable for reconnect 2025-06-29 03:51:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f2fe293b]) (logid:232bdc06) Completed resetting hosts suitable for reconnect 2025-06-29 03:51:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f2fe293b]) (logid:232bdc06) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:51:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f2fe293b]) (logid:232bdc06) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:51:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f2fe293b]) (logid:232bdc06) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:51:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f2fe293b]) (logid:232bdc06) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:51:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9470f3ea]) (logid:d9e16d0f) VM scheduler.poll is being called at 2025-06-29T01:52:00+0000 2025-06-29 03:51:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9470f3ea]) (logid:d9e16d0f) Got 0 scheduled jobs to be executed at 2025-06-29T01:52:00+0000 2025-06-29 03:51:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9470f3ea]) (logid:d9e16d0f) Cleaned up 0 VM scheduled job entries 2025-06-29 03:51:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9a9049e3]) (logid:e062cd85) VM Operation Thread Running 2025-06-29 03:51:48,366 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-dfcab92c]) (logid:c980d556) HA health check task is running... 2025-06-29 03:51:49,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67df59d5]) (logid:6cd25b32) No inactive management server node found 2025-06-29 03:51:49,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67df59d5]) (logid:6cd25b32) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9a5a352]) (logid:32ecceee) No inactive management server node found 2025-06-29 03:51:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9a5a352]) (logid:32ecceee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:52,368 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-014b1c24]) (logid:a1929015) HA health check task is running... 2025-06-29 03:51:52,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72da06c5]) (logid:6da25f69) No inactive management server node found 2025-06-29 03:51:52,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72da06c5]) (logid:6da25f69) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-90dd0c88]) (logid:0ed8aa22) No inactive management server node found 2025-06-29 03:51:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-90dd0c88]) (logid:0ed8aa22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:55,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8355: Seq 7-8355: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:51:55,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8355 2025-06-29 03:51:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebaa0f31]) (logid:28cd7218) No inactive management server node found 2025-06-29 03:51:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebaa0f31]) (logid:28cd7218) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:56,369 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a0d38f66]) (logid:3770fc4a) HA health check task is running... 2025-06-29 03:51:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d938334]) (logid:00007ea0) No inactive management server node found 2025-06-29 03:51:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d938334]) (logid:00007ea0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:58,086 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-8e7e3429]) (logid:cb8fc911) AutoScaling Monitor is running... 2025-06-29 03:51:58,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-aef01ee5]) (logid:f53d3fb9) Enabled non-edge zones available for scan: 1 2025-06-29 03:51:58,091 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-00c4b6cd]) (logid:0e556458) Starting to remove all volume_stats rows older than [Sat Jun 28 15:51:58 CEST 2025]. 2025-06-29 03:51:58,091 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-00c4b6cd]) (logid:0e556458) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:51:58 CEST 2025]. 2025-06-29 03:51:58,097 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-aef01ee5]) (logid:f53d3fb9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:51:58,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-aef01ee5]) (logid:f53d3fb9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:51:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-71ab692f]) (logid:5204fb97) Enabled non-edge zones available for scan: 1 2025-06-29 03:51:58,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-71ab692f]) (logid:5204fb97) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:51:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9171d6a8]) (logid:f01ad162) No inactive management server node found 2025-06-29 03:51:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9171d6a8]) (logid:f01ad162) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:51:58,517 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-f45aa63c]) (logid:c58c3a54) Starting to remove all vm_stats rows older than [Sat Jun 28 15:51:58 CEST 2025]. 2025-06-29 03:51:58,550 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-f45aa63c]) (logid:c58c3a54) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:51:58 CEST 2025]. 2025-06-29 03:51:59,230 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:51:59,232 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:51:59,234 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) Seq 2-4725683383995203654: Routed from 90520738888109 2025-06-29 03:51:59,278 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) Seq 1-4725683383995203654: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:51:59,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:51:59,280 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) Seq 2-3926857400090361936: Routed from 90520738888109 2025-06-29 03:51:59,326 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-cb696cfa]) (logid:675223ae) Seq 2-3926857400090361936: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:51:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b789f90]) (logid:55d6ab13) No inactive management server node found 2025-06-29 03:51:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b789f90]) (logid:55d6ab13) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:00,371 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-512131fb]) (logid:a3da8865) HA health check task is running... 2025-06-29 03:52:01,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bf11d3c]) (logid:8897ffe5) No inactive management server node found 2025-06-29 03:52:01,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bf11d3c]) (logid:8897ffe5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:02,252 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:52:02,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:52:02,259 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) Seq 2-4725683383995203655: Routed from 90520738888109 2025-06-29 03:52:02,314 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) Seq 1-4725683383995203655: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:52:02,360 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:52:02,361 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) Seq 2-3926857400090361937: Routed from 90520738888109 2025-06-29 03:52:02,459 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-98bcdafc]) (logid:31e84f5c) Seq 2-3926857400090361937: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:52:02,724 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) StorageCollector is running... 2025-06-29 03:52:02,729 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:52:02,729 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:52:02,729 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:52:02,730 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-3938116399158788119: Routed from 90520738888109 2025-06-29 03:52:02,785 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 4-3938116399158788119: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:52:02,789 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:52:02,789 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:52:02,789 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:52:02,790 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-3926857400090361938: Routed from 90520738888109 2025-06-29 03:52:02,801 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-3926857400090361938: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:52:02,824 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:52:02,824 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:52:02,824 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:52:02,825 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-4725683383995203656: Routed from 90520738888109 2025-06-29 03:52:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ece285e9]) (logid:4af3eb94) No inactive management server node found 2025-06-29 03:52:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ece285e9]) (logid:4af3eb94) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:02,873 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 1-4725683383995203656: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:52:02,883 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:52:02,883 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:52:02,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:52:02,884 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-4725683383995203657: Routed from 90520738888109 2025-06-29 03:52:02,969 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 1-4725683383995203657: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:52:02,971 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:52:02,971 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:52:02,971 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:52:02,972 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-3926857400090361939: Routed from 90520738888109 2025-06-29 03:52:03,017 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1ac0cc0d]) (logid:5e581b35) Seq 2-3926857400090361939: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:52:04,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c704dec0]) (logid:be4a15e4) No inactive management server node found 2025-06-29 03:52:04,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c704dec0]) (logid:be4a15e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:04,372 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e8cf2fcc]) (logid:8f567b52) HA health check task is running... 2025-06-29 03:52:05,248 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8356: Seq 7-8356: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:52:05,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8356 2025-06-29 03:52:05,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc192c66]) (logid:f7cdfa79) No inactive management server node found 2025-06-29 03:52:05,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc192c66]) (logid:f7cdfa79) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1559eede]) (logid:7bcbf2ef) No inactive management server node found 2025-06-29 03:52:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1559eede]) (logid:7bcbf2ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:08,373 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-978d5e23]) (logid:8bae0625) HA health check task is running... 2025-06-29 03:52:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdfa96af]) (logid:a8bcccd4) No inactive management server node found 2025-06-29 03:52:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdfa96af]) (logid:a8bcccd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:10,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae398c11]) (logid:1423a266) No inactive management server node found 2025-06-29 03:52:10,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae398c11]) (logid:1423a266) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ae6b324]) (logid:3f89cf87) No inactive management server node found 2025-06-29 03:52:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ae6b324]) (logid:3f89cf87) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:12,374 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-eeff680b]) (logid:4759f4b3) HA health check task is running... 2025-06-29 03:52:13,002 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-9a33e544]) (logid:9d11753c) Checking if any host reservation can be released ... 2025-06-29 03:52:13,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-9a33e544]) (logid:9d11753c) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:13,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-9a33e544]) (logid:9d11753c) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-9a33e544]) (logid:9d11753c) Done running HostReservationReleaseChecker ... 2025-06-29 03:52:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-15c155c5]) (logid:246838e0) Found 0 routers to update status. 2025-06-29 03:52:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-15c155c5]) (logid:246838e0) Found 0 VPC's to update Redundant State. 2025-06-29 03:52:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-15c155c5]) (logid:246838e0) Found 0 networks to update RvR status. 2025-06-29 03:52:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Running Capacity Checker ... 2025-06-29 03:52:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) recalculating system capacity 2025-06-29 03:52:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Executing cpu/ram capacity update 2025-06-29 03:52:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:13,173 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:52:13,192 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:52:13,192 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:52:13,205 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:13,207 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:13,214 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:52:13,217 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:52:13,218 DEBUG [c.c.c.CapacityManagerImpl] (pool-18-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:52:13,228 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Done executing cpu/ram capacity update 2025-06-29 03:52:13,228 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Executing storage capacity update 2025-06-29 03:52:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:52:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:52:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:52:13,244 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:52:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:52:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:52:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:52:13,256 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:52:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:52:13,265 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:52:13,265 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:52:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:52:13,281 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:52:13,281 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:52:13,281 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:52:13,286 DEBUG [c.c.s.StorageManagerImpl] (pool-19-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:52:13,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Done executing storage capacity update 2025-06-29 03:52:13,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Executing capacity updates for public ip and Vlans 2025-06-29 03:52:13,309 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Done capacity updates for public ip and Vlans 2025-06-29 03:52:13,309 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Executing capacity updates for private ip 2025-06-29 03:52:13,313 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Done executing capacity updates for private ip 2025-06-29 03:52:13,313 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Done recalculating system capacity 2025-06-29 03:52:13,360 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5dc670c4]) (logid:b6511c8e) Done running Capacity Checker ... 2025-06-29 03:52:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2f885e4]) (logid:ee9d1dc0) No inactive management server node found 2025-06-29 03:52:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2f885e4]) (logid:ee9d1dc0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:52:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b2ffae3]) (logid:2f35823a) No inactive management server node found 2025-06-29 03:52:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b2ffae3]) (logid:2f35823a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:15,247 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8357: Seq 7-8357: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:52:15,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8357 2025-06-29 03:52:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74f75ae7]) (logid:5b04c94c) No inactive management server node found 2025-06-29 03:52:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74f75ae7]) (logid:5b04c94c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:16,376 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-88396c4f]) (logid:6f1a5106) HA health check task is running... 2025-06-29 03:52:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47259088]) (logid:fab4887d) No inactive management server node found 2025-06-29 03:52:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47259088]) (logid:fab4887d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05b773df]) (logid:43fa44ed) No inactive management server node found 2025-06-29 03:52:19,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05b773df]) (logid:43fa44ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:20,377 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b26f25df]) (logid:9f8f1dc3) HA health check task is running... 2025-06-29 03:52:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61b54364]) (logid:fd6073da) No inactive management server node found 2025-06-29 03:52:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61b54364]) (logid:fd6073da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a991bea9]) (logid:bf854525) No inactive management server node found 2025-06-29 03:52:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a991bea9]) (logid:bf854525) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:23,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac8a93ff]) (logid:3a65e838) No inactive management server node found 2025-06-29 03:52:23,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac8a93ff]) (logid:3a65e838) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:24,378 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0e951c71]) (logid:88b1d046) HA health check task is running... 2025-06-29 03:52:25,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8358: Seq 7-8358: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:52:25,271 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8358 2025-06-29 03:52:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7ff9e20]) (logid:70a92abd) No inactive management server node found 2025-06-29 03:52:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7ff9e20]) (logid:70a92abd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36c5b896]) (logid:7e0840c0) No inactive management server node found 2025-06-29 03:52:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36c5b896]) (logid:7e0840c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-63b4eb9e]) (logid:3a425f1d) Enabled non-edge zones available for scan: 1 2025-06-29 03:52:28,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-63b4eb9e]) (logid:3a425f1d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:52:28,100 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-63b4eb9e]) (logid:3a425f1d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:52:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-28f76660]) (logid:07d87d6f) Enabled non-edge zones available for scan: 1 2025-06-29 03:52:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-28f76660]) (logid:07d87d6f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:52:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6e20c94]) (logid:4855a4a1) No inactive management server node found 2025-06-29 03:52:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6e20c94]) (logid:4855a4a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:28,380 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8b86e095]) (logid:3258c79c) HA health check task is running... 2025-06-29 03:52:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a01d0fc]) (logid:3c41dfde) No inactive management server node found 2025-06-29 03:52:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a01d0fc]) (logid:3c41dfde) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-576c6b3f]) (logid:4badb460) No inactive management server node found 2025-06-29 03:52:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-576c6b3f]) (logid:4badb460) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:32,381 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1cd18721]) (logid:d0b4bc02) HA health check task is running... 2025-06-29 03:52:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d1562ac]) (logid:5c909574) No inactive management server node found 2025-06-29 03:52:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d1562ac]) (logid:5c909574) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c907fe96]) (logid:f981bd55) No inactive management server node found 2025-06-29 03:52:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c907fe96]) (logid:f981bd55) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:35,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8359: Seq 7-8359: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:52:35,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8359 2025-06-29 03:52:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be331c0f]) (logid:99c970c6) No inactive management server node found 2025-06-29 03:52:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be331c0f]) (logid:99c970c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:36,382 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-187236b5]) (logid:844c20df) HA health check task is running... 2025-06-29 03:52:37,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-399d26e5]) (logid:70edc06d) No inactive management server node found 2025-06-29 03:52:37,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-399d26e5]) (logid:70edc06d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e585aa]) (logid:3beaaa19) No inactive management server node found 2025-06-29 03:52:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e585aa]) (logid:3beaaa19) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:40,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5ddf89a]) (logid:c6802c0e) No inactive management server node found 2025-06-29 03:52:40,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5ddf89a]) (logid:c6802c0e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:40,384 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3e0d2ff9]) (logid:4e1ada83) HA health check task is running... 2025-06-29 03:52:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d1f2509]) (logid:b0d8f2c2) No inactive management server node found 2025-06-29 03:52:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d1f2509]) (logid:b0d8f2c2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-0d808c68]) (logid:5f3e8f13) DbCollector is running... 2025-06-29 03:52:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7fe8bc03]) (logid:10961593) Found 0 routers to update status. 2025-06-29 03:52:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7fe8bc03]) (logid:10961593) Found 0 VPC's to update Redundant State. 2025-06-29 03:52:43,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7fe8bc03]) (logid:10961593) Found 0 networks to update RvR status. 2025-06-29 03:52:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-986fa732]) (logid:8f0a1100) No inactive management server node found 2025-06-29 03:52:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-986fa732]) (logid:8f0a1100) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) ManagementServerCollector is running... 2025-06-29 03:52:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) DB properties were already loaded 2025-06-29 03:52:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:52:44,054 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Execution of process [4065497] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:52:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Exit value of process [4065497] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:52:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Process [4065497] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:52:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) usage install: null 2025-06-29 03:52:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Sessions active: 0 2025-06-29 03:52:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:52:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065500] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:52:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:52:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065505] for command [/bin/bash -c uname -r ]. 2025-06-29 03:52:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:52:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065506] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:52:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) system memory from /proc: 4887515136 2025-06-29 03:52:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:52:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065511] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:52:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) free memory from /proc: 187011072 2025-06-29 03:52:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:52:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065516] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:52:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) used memory from /proc: 1073004 2025-06-29 03:52:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:52:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065517] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:52:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:52:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065518] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:52:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:52:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065519] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:52:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:52:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065526] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:52:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:52:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Successfully executed process [4065527] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail buffersdirect.capacity, 242617 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail buffersdirect.count, 75 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail buffersdirect.used, 242617 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail buffersmapped.capacity, 0 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail buffersmapped.count, 0 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail buffersmapped.used, 0 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:52:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail jvmuptime, 1023834 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memoryheap.committed, 545259520 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memoryheap.init, 77594624 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memoryheap.usage, 0.2447447657617885 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorynon-heap.committed, 191168512 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorynon-heap.max, -1 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorynon-heap.usage, -1.8631876E8 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorynon-heap.used, 186318760 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09596399293521728 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19062843669810378 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.Compressed-Class-Space.usage, 0.012211471796035767 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.Metaspace.usage, 0.993497421818983 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.PS-Eden-Space.usage, 0.42789373059756286 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorytotal.committed, 736428032 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorytotal.init, 85262336 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorytotal.max, 1908932607 2025-06-29 03:52:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail memorytotal.used, 653889624 2025-06-29 03:52:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail threadsdeadlocks, [] 2025-06-29 03:52:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail threadsnew.count, 0 2025-06-29 03:52:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:52:44,135 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-025716ad]) (logid:9c3d68b4) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":79091392,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1023760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.28,"totalInit":85262336,"totalUsed":652476184,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":467201264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":187011072,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.28,0.17,0.17],"systemCyclesUsage":[66400733,53824745,4589012192],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:52:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 33, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":79091392,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1023760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.28,"totalInit":85262336,"totalUsed":652476184,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":467201264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":187011072,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.28,0.17,0.17],"systemCyclesUsage":[66400733,53824745,4589012192],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:52:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:52:44,142 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-11:[ctx-2adceaff]) (logid:033451cb) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":79091392,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1023760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.28,"totalInit":85262336,"totalUsed":652476184,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":467201264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":187011072,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.28,0.17,0.17],"systemCyclesUsage":[66400733,53824745,4589012192],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:52:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 03:52:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 33, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":79091392,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1023760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.28,"totalInit":85262336,"totalUsed":652476184,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":467201264,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":187011072,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.28,0.17,0.17],"systemCyclesUsage":[66400733,53824745,4589012192],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:52:44,385 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-aab43f34]) (logid:0a987cf1) HA health check task is running... 2025-06-29 03:52:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a64f900]) (logid:bb1ebb3d) No inactive management server node found 2025-06-29 03:52:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a64f900]) (logid:bb1ebb3d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:45,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8360: Seq 7-8360: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:52:45,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8360 2025-06-29 03:52:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:52:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:52:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66d0df44]) (logid:b412ee0a) No inactive management server node found 2025-06-29 03:52:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66d0df44]) (logid:b412ee0a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:46,649 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:46,652 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:52:46,655 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:52:47,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:47,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:52:47,083 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:52:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5afd75d4]) (logid:68d3fbfd) No inactive management server node found 2025-06-29 03:52:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5afd75d4]) (logid:68d3fbfd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-134a3567]) (logid:2cca4b75) ClusterDRS.poll is being called at 2025-06-29T01:53:00+0000 2025-06-29 03:52:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-134a3567]) (logid:2cca4b75) Removed 0 old drs migration plans 2025-06-29 03:52:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bdbc6bb5]) (logid:8be2d6a5) VM scheduler.poll is being called at 2025-06-29T01:53:00+0000 2025-06-29 03:52:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bdbc6bb5]) (logid:8be2d6a5) Got 0 scheduled jobs to be executed at 2025-06-29T01:53:00+0000 2025-06-29 03:52:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bdbc6bb5]) (logid:8be2d6a5) Cleaned up 0 VM scheduled job entries 2025-06-29 03:52:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-8123abda]) (logid:5bf7090a) VM Operation Thread Running 2025-06-29 03:52:48,387 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-aed8d095]) (logid:2d5af1ac) HA health check task is running... 2025-06-29 03:52:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10aa9e6e]) (logid:0bcd8a6a) No inactive management server node found 2025-06-29 03:52:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10aa9e6e]) (logid:0bcd8a6a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58778351]) (logid:03ef7f86) No inactive management server node found 2025-06-29 03:52:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58778351]) (logid:03ef7f86) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1daf4583]) (logid:c75a42dd) No inactive management server node found 2025-06-29 03:52:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1daf4583]) (logid:c75a42dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:52,388 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-536cb9b2]) (logid:ca040492) HA health check task is running... 2025-06-29 03:52:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c704a6cb]) (logid:80eded1e) No inactive management server node found 2025-06-29 03:52:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c704a6cb]) (logid:80eded1e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:52:55,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8362: Seq 7-8362: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:52:55,271 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8362 2025-06-29 03:52:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29487029]) (logid:c95548f9) No inactive management server node found 2025-06-29 03:52:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29487029]) (logid:c95548f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:56,389 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-659c9e64]) (logid:3dc191c9) HA health check task is running... 2025-06-29 03:52:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dc719ba]) (logid:15ebf8e1) No inactive management server node found 2025-06-29 03:52:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dc719ba]) (logid:15ebf8e1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ffea07d7]) (logid:3d5d0ebf) Enabled non-edge zones available for scan: 1 2025-06-29 03:52:58,090 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-c63e0616]) (logid:0701aa02) AutoScaling Monitor is running... 2025-06-29 03:52:58,093 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-a5a4b931]) (logid:d2756a20) Starting to remove all volume_stats rows older than [Sat Jun 28 15:52:58 CEST 2025]. 2025-06-29 03:52:58,094 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-a5a4b931]) (logid:d2756a20) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:52:58 CEST 2025]. 2025-06-29 03:52:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ffea07d7]) (logid:3d5d0ebf) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:52:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ffea07d7]) (logid:3d5d0ebf) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:52:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-036670f9]) (logid:238d0119) Enabled non-edge zones available for scan: 1 2025-06-29 03:52:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-036670f9]) (logid:238d0119) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:52:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d941714]) (logid:5d3d458c) No inactive management server node found 2025-06-29 03:52:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d941714]) (logid:5d3d458c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:52:58,551 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-a90f8ac7]) (logid:181439c0) Starting to remove all vm_stats rows older than [Sat Jun 28 15:52:58 CEST 2025]. 2025-06-29 03:52:58,574 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-a90f8ac7]) (logid:181439c0) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:52:58 CEST 2025]. 2025-06-29 03:52:59,330 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:52:59,332 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:52:59,334 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) Seq 2-4725683383995203658: Routed from 90520738888109 2025-06-29 03:52:59,378 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) Seq 1-4725683383995203658: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:52:59,379 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:52:59,380 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) Seq 2-3926857400090361940: Routed from 90520738888109 2025-06-29 03:52:59,426 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a934b9d5]) (logid:2e3036d6) Seq 2-3926857400090361940: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:52:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c6435c9]) (logid:599618e3) No inactive management server node found 2025-06-29 03:52:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c6435c9]) (logid:599618e3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:00,250 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8363: Seq 7-8363: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:00,271 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8363 2025-06-29 03:53:00,391 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5ab8f7d5]) (logid:0a93fe46) HA health check task is running... 2025-06-29 03:53:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-676bc811]) (logid:baf06e64) No inactive management server node found 2025-06-29 03:53:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-676bc811]) (logid:baf06e64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:02,514 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:53:02,518 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:53:02,519 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) Seq 2-4725683383995203659: Routed from 90520738888109 2025-06-29 03:53:02,574 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) Seq 1-4725683383995203659: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:53:02,610 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:53:02,611 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) Seq 2-3926857400090361941: Routed from 90520738888109 2025-06-29 03:53:02,698 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0de31dcd]) (logid:43594c2e) Seq 2-3926857400090361941: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:53:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10753fd3]) (logid:863d170b) No inactive management server node found 2025-06-29 03:53:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10753fd3]) (logid:863d170b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:03,017 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) StorageCollector is running... 2025-06-29 03:53:03,022 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:53:03,022 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:53:03,022 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:53:03,023 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-3938116399158788120: Routed from 90520738888109 2025-06-29 03:53:03,084 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 4-3938116399158788120: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:53:03,088 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:53:03,088 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:53:03,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:53:03,089 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-3926857400090361942: Routed from 90520738888109 2025-06-29 03:53:03,101 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-3926857400090361942: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:53:03,113 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:53:03,113 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:53:03,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:53:03,114 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-4725683383995203660: Routed from 90520738888109 2025-06-29 03:53:03,162 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 1-4725683383995203660: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:53:03,174 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:53:03,174 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:53:03,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:53:03,175 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-4725683383995203661: Routed from 90520738888109 2025-06-29 03:53:03,259 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 1-4725683383995203661: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:53:03,261 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:53:03,261 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:53:03,261 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:53:03,262 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-3926857400090361943: Routed from 90520738888109 2025-06-29 03:53:03,307 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2bc9d151]) (logid:c1495390) Seq 2-3926857400090361943: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:53:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23329e81]) (logid:f127c0f4) No inactive management server node found 2025-06-29 03:53:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23329e81]) (logid:f127c0f4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:04,392 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c5111dfe]) (logid:7c63dbeb) HA health check task is running... 2025-06-29 03:53:05,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-507b12ba]) (logid:5b9ebcb5) No inactive management server node found 2025-06-29 03:53:05,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-507b12ba]) (logid:5b9ebcb5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f988b1]) (logid:b5a0e757) No inactive management server node found 2025-06-29 03:53:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f988b1]) (logid:b5a0e757) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:08,393 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e65791bc]) (logid:aa7bad21) HA health check task is running... 2025-06-29 03:53:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-085b7b4e]) (logid:d5a020e0) No inactive management server node found 2025-06-29 03:53:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-085b7b4e]) (logid:d5a020e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:10,250 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8364: Seq 7-8364: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:10,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8364 2025-06-29 03:53:10,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d81c3ccc]) (logid:357a5ee3) No inactive management server node found 2025-06-29 03:53:10,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d81c3ccc]) (logid:357a5ee3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-950722c5]) (logid:31dd92ac) No inactive management server node found 2025-06-29 03:53:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-950722c5]) (logid:31dd92ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:12,395 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6db99cf4]) (logid:327bd231) HA health check task is running... 2025-06-29 03:53:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dcdbf110]) (logid:03d2ca38) Found 0 routers to update status. 2025-06-29 03:53:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dcdbf110]) (logid:03d2ca38) Found 0 VPC's to update Redundant State. 2025-06-29 03:53:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dcdbf110]) (logid:03d2ca38) Found 0 networks to update RvR status. 2025-06-29 03:53:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7628fef5]) (logid:09aafe5a) No inactive management server node found 2025-06-29 03:53:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7628fef5]) (logid:09aafe5a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b64d08d9]) (logid:f28ff982) No inactive management server node found 2025-06-29 03:53:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b64d08d9]) (logid:f28ff982) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c9b2bda]) (logid:8362fb66) No inactive management server node found 2025-06-29 03:53:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c9b2bda]) (logid:8362fb66) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:16,396 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8c3f7f9d]) (logid:4e1d0a28) HA health check task is running... 2025-06-29 03:53:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d336f39a]) (logid:f3ea2723) No inactive management server node found 2025-06-29 03:53:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d336f39a]) (logid:f3ea2723) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:18,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-57b74785]) (logid:7cd9410c) Resetting hosts suitable for reconnect 2025-06-29 03:53:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-57b74785]) (logid:7cd9410c) Completed resetting hosts suitable for reconnect 2025-06-29 03:53:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-57b74785]) (logid:7cd9410c) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:53:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-57b74785]) (logid:7cd9410c) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:53:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-57b74785]) (logid:7cd9410c) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:53:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-57b74785]) (logid:7cd9410c) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:53:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65a63e7]) (logid:d2681a1c) No inactive management server node found 2025-06-29 03:53:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65a63e7]) (logid:d2681a1c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:20,250 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8365: Seq 7-8365: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:20,261 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8365 2025-06-29 03:53:20,397 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cd418a97]) (logid:fc98a6ae) HA health check task is running... 2025-06-29 03:53:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ee82f83]) (logid:f33d9e1d) No inactive management server node found 2025-06-29 03:53:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ee82f83]) (logid:f33d9e1d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:22,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ba1c15a]) (logid:58ce07bf) No inactive management server node found 2025-06-29 03:53:22,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ba1c15a]) (logid:58ce07bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:23,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68054402]) (logid:22340aad) No inactive management server node found 2025-06-29 03:53:23,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68054402]) (logid:22340aad) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:53:24,398 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d358dd0f]) (logid:67977677) HA health check task is running... 2025-06-29 03:53:25,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41acb4cc]) (logid:4342ed90) No inactive management server node found 2025-06-29 03:53:25,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41acb4cc]) (logid:4342ed90) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edb6e42d]) (logid:511b7768) No inactive management server node found 2025-06-29 03:53:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edb6e42d]) (logid:511b7768) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-09a1f117]) (logid:8668c610) Enabled non-edge zones available for scan: 1 2025-06-29 03:53:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-09a1f117]) (logid:8668c610) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:53:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-09a1f117]) (logid:8668c610) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:53:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4292febb]) (logid:f39ee92f) Enabled non-edge zones available for scan: 1 2025-06-29 03:53:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4292febb]) (logid:f39ee92f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:53:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dc7b4de]) (logid:0dfa8a39) No inactive management server node found 2025-06-29 03:53:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dc7b4de]) (logid:0dfa8a39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:28,400 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-99c3bc45]) (logid:187c4b86) HA health check task is running... 2025-06-29 03:53:29,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b6d547c]) (logid:065246c5) No inactive management server node found 2025-06-29 03:53:29,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b6d547c]) (logid:065246c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:30,252 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8366: Seq 7-8366: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:30,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8366 2025-06-29 03:53:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71d7e0c0]) (logid:10c23028) No inactive management server node found 2025-06-29 03:53:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71d7e0c0]) (logid:10c23028) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:32,401 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6b9dffc4]) (logid:61dba0a0) HA health check task is running... 2025-06-29 03:53:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac1b10a2]) (logid:df1a1557) No inactive management server node found 2025-06-29 03:53:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac1b10a2]) (logid:df1a1557) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7980c477]) (logid:d7756a5f) No inactive management server node found 2025-06-29 03:53:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7980c477]) (logid:d7756a5f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:35,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92229c18]) (logid:63eca436) No inactive management server node found 2025-06-29 03:53:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92229c18]) (logid:63eca436) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:36,403 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-be755e84]) (logid:601678c8) HA health check task is running... 2025-06-29 03:53:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a40e028]) (logid:463c6674) No inactive management server node found 2025-06-29 03:53:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a40e028]) (logid:463c6674) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c770df00]) (logid:b6994dfb) No inactive management server node found 2025-06-29 03:53:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c770df00]) (logid:b6994dfb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:40,251 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8367: Seq 7-8367: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:40,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8367 2025-06-29 03:53:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bedd4b6]) (logid:c73ed64b) No inactive management server node found 2025-06-29 03:53:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bedd4b6]) (logid:c73ed64b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:40,404 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-aaf02c3a]) (logid:0a7f7d3c) HA health check task is running... 2025-06-29 03:53:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-056061d6]) (logid:89e5344e) No inactive management server node found 2025-06-29 03:53:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-056061d6]) (logid:89e5344e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-303111c6]) (logid:fe1f41f8) DbCollector is running... 2025-06-29 03:53:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad4a4d89]) (logid:06a53818) Found 0 routers to update status. 2025-06-29 03:53:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad4a4d89]) (logid:06a53818) Found 0 VPC's to update Redundant State. 2025-06-29 03:53:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad4a4d89]) (logid:06a53818) Found 0 networks to update RvR status. 2025-06-29 03:53:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5034cc51]) (logid:ce45b6dc) No inactive management server node found 2025-06-29 03:53:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5034cc51]) (logid:ce45b6dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) ManagementServerCollector is running... 2025-06-29 03:53:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) DB properties were already loaded 2025-06-29 03:53:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:53:44,059 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Execution of process [4065532] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:53:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Exit value of process [4065532] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:53:44,060 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Process [4065532] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:53:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) usage install: null 2025-06-29 03:53:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Sessions active: 0 2025-06-29 03:53:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:53:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065535] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:53:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:53:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065540] for command [/bin/bash -c uname -r ]. 2025-06-29 03:53:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:53:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065541] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:53:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) system memory from /proc: 4887515136 2025-06-29 03:53:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:53:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065546] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:53:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) free memory from /proc: 189816832 2025-06-29 03:53:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:53:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065551] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:53:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) used memory from /proc: 1073004 2025-06-29 03:53:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:53:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065552] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:53:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:53:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065553] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:53:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:53:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065554] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:53:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:53:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065561] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:53:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:53:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Successfully executed process [4065562] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail buffersdirect.capacity, 242726 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail buffersdirect.count, 75 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail buffersdirect.used, 242726 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail buffersmapped.capacity, 0 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail buffersmapped.count, 0 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail buffersmapped.used, 0 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail jvmuptime, 1083832 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memoryheap.committed, 545259520 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memoryheap.init, 77594624 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memoryheap.usage, 0.2565675948681788 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorynon-heap.committed, 191168512 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorynon-heap.max, -1 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorynon-heap.usage, -1.86518856E8 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorynon-heap.used, 186518856 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09658570214609437 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19146157563235244 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.Compressed-Class-Space.usage, 0.012212499976158142 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.Metaspace.usage, 0.9936445830656967 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.PS-Eden-Space.usage, 0.46326427035162693 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorytotal.committed, 736428032 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorytotal.init, 85262336 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorytotal.max, 1908932607 2025-06-29 03:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail memorytotal.used, 676853400 2025-06-29 03:53:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail threadsdeadlocks, [] 2025-06-29 03:53:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail threadsnew.count, 0 2025-06-29 03:53:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) not storing detail threadstimed_waiting.count, 82 2025-06-29 03:53:44,134 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-80c7627d]) (logid:2a06c42a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":56087256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1083763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.57,"totalInit":85262336,"totalUsed":675687792,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":489770248,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":189816832,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.57,0.27,0.21],"systemCyclesUsage":[66401090,53825040,4589035287],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:53:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 35, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":56087256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1083763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.57,"totalInit":85262336,"totalUsed":675687792,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":489770248,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":189816832,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.57,0.27,0.21],"systemCyclesUsage":[66401090,53825040,4589035287],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:53:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:53:44,141 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-11:[ctx-f50046d5]) (logid:ab06e27c) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":56087256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1083763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.57,"totalInit":85262336,"totalUsed":675687792,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":489770248,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":189816832,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.57,0.27,0.21],"systemCyclesUsage":[66401090,53825040,4589035287],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:53:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 03:53:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 35, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":56087256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1083763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.57,"totalInit":85262336,"totalUsed":675687792,"totalCommitted":736428032,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":489770248,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":197,"threadsWaitingCount":105,"systemMemoryTotal":4887515136,"systemMemoryFree":189816832,"systemMemoryUsed":1073004,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.57,0.27,0.21],"systemCyclesUsage":[66401090,53825040,4589035287],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:53:44,405 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7040cd09]) (logid:5aa083fe) HA health check task is running... 2025-06-29 03:53:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db341edd]) (logid:b4d10bfd) No inactive management server node found 2025-06-29 03:53:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db341edd]) (logid:b4d10bfd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:45,252 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8368: Seq 7-8368: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:45,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8368 2025-06-29 03:53:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:53:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:53:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b53f2b3]) (logid:0d800d10) No inactive management server node found 2025-06-29 03:53:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b53f2b3]) (logid:0d800d10) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:53:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:53:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:53:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:53:47,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:53:47,094 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:53:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:53:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:53:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e7b46c9]) (logid:0c810536) No inactive management server node found 2025-06-29 03:53:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e7b46c9]) (logid:0c810536) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1e26ea39]) (logid:3f74ecb9) ClusterDRS.poll is being called at 2025-06-29T01:54:00+0000 2025-06-29 03:53:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1e26ea39]) (logid:3f74ecb9) Removed 0 old drs migration plans 2025-06-29 03:53:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-270189e6]) (logid:a327b01f) VM scheduler.poll is being called at 2025-06-29T01:54:00+0000 2025-06-29 03:53:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-270189e6]) (logid:a327b01f) Got 0 scheduled jobs to be executed at 2025-06-29T01:54:00+0000 2025-06-29 03:53:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-270189e6]) (logid:a327b01f) Cleaned up 0 VM scheduled job entries 2025-06-29 03:53:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-cd23b476]) (logid:b8211f20) VM Operation Thread Running 2025-06-29 03:53:48,407 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d54a0e5b]) (logid:5b78ad33) HA health check task is running... 2025-06-29 03:53:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db4a89b3]) (logid:16cd3268) No inactive management server node found 2025-06-29 03:53:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db4a89b3]) (logid:16cd3268) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8974dce1]) (logid:5093e9e8) No inactive management server node found 2025-06-29 03:53:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8974dce1]) (logid:5093e9e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:52,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ee860aa]) (logid:4e8d7153) No inactive management server node found 2025-06-29 03:53:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ee860aa]) (logid:4e8d7153) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:52,409 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4bf6adb0]) (logid:3e667fd5) HA health check task is running... 2025-06-29 03:53:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77e320e6]) (logid:5b1fe23d) No inactive management server node found 2025-06-29 03:53:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77e320e6]) (logid:5b1fe23d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:55,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8370: Seq 7-8370: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:53:55,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8370 2025-06-29 03:53:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-247bb64d]) (logid:648b9451) No inactive management server node found 2025-06-29 03:53:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-247bb64d]) (logid:648b9451) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:56,410 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-934fbcf4]) (logid:811afd85) HA health check task is running... 2025-06-29 03:53:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36885c05]) (logid:5cf9bae4) No inactive management server node found 2025-06-29 03:53:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36885c05]) (logid:5cf9bae4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-530614cb]) (logid:83eaa00a) Enabled non-edge zones available for scan: 1 2025-06-29 03:53:58,092 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-0081c7ee]) (logid:a51f034a) AutoScaling Monitor is running... 2025-06-29 03:53:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-530614cb]) (logid:83eaa00a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:53:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-530614cb]) (logid:83eaa00a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:53:58,096 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-49822413]) (logid:4645dfae) Starting to remove all volume_stats rows older than [Sat Jun 28 15:53:58 CEST 2025]. 2025-06-29 03:53:58,097 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-49822413]) (logid:4645dfae) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:53:58 CEST 2025]. 2025-06-29 03:53:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-34afbcce]) (logid:29934fc0) Enabled non-edge zones available for scan: 1 2025-06-29 03:53:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-34afbcce]) (logid:29934fc0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:53:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbc8e4d6]) (logid:da83a9ed) No inactive management server node found 2025-06-29 03:53:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbc8e4d6]) (logid:da83a9ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:53:58,576 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-742cf611]) (logid:94382ad5) Starting to remove all vm_stats rows older than [Sat Jun 28 15:53:58 CEST 2025]. 2025-06-29 03:53:58,601 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-742cf611]) (logid:94382ad5) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:53:58 CEST 2025]. 2025-06-29 03:53:59,429 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:53:59,431 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:53:59,432 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) Seq 2-4725683383995203662: Routed from 90520738888109 2025-06-29 03:53:59,476 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) Seq 1-4725683383995203662: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:53:59,477 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:53:59,478 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) Seq 2-3926857400090361944: Routed from 90520738888109 2025-06-29 03:53:59,523 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-a3b70616]) (logid:19100dd5) Seq 2-3926857400090361944: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:53:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8839fef]) (logid:65b2e5cb) No inactive management server node found 2025-06-29 03:53:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8839fef]) (logid:65b2e5cb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:00,412 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-540bf06f]) (logid:62be57fe) HA health check task is running... 2025-06-29 03:54:01,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f6424c4]) (logid:3e3bad8a) No inactive management server node found 2025-06-29 03:54:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f6424c4]) (logid:3e3bad8a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:02,754 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:54:02,756 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:54:02,757 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) Seq 2-4725683383995203663: Routed from 90520738888109 2025-06-29 03:54:02,810 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) Seq 1-4725683383995203663: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:54:02,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ddf9902]) (logid:3107d271) No inactive management server node found 2025-06-29 03:54:02,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ddf9902]) (logid:3107d271) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:02,862 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:54:02,863 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) Seq 2-3926857400090361945: Routed from 90520738888109 2025-06-29 03:54:02,955 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-984abec6]) (logid:4db10ba2) Seq 2-3926857400090361945: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:54:03,307 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) StorageCollector is running... 2025-06-29 03:54:03,317 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:54:03,317 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:54:03,318 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:54:03,320 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-3938116399158788121: Routed from 90520738888109 2025-06-29 03:54:03,378 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 4-3938116399158788121: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:54:03,384 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:54:03,384 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:54:03,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:54:03,386 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-4725683383995203664: Routed from 90520738888109 2025-06-29 03:54:03,434 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 1-4725683383995203664: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:54:03,457 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:54:03,457 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:54:03,457 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:54:03,458 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-3926857400090361946: Routed from 90520738888109 2025-06-29 03:54:03,471 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-3926857400090361946: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:54:03,487 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:54:03,487 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:54:03,487 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:54:03,489 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-4725683383995203665: Routed from 90520738888109 2025-06-29 03:54:03,533 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 1-4725683383995203665: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:54:03,535 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:54:03,535 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:54:03,535 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:54:03,536 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-3926857400090361947: Routed from 90520738888109 2025-06-29 03:54:03,580 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-669f444e]) (logid:72d58573) Seq 2-3926857400090361947: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:54:04,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-918feeef]) (logid:c6c76fd3) No inactive management server node found 2025-06-29 03:54:04,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-918feeef]) (logid:c6c76fd3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:04,413 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-526439ae]) (logid:bd24a9a8) HA health check task is running... 2025-06-29 03:54:05,253 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8371: Seq 7-8371: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:54:05,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8371 2025-06-29 03:54:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0724b1b8]) (logid:5c91169b) No inactive management server node found 2025-06-29 03:54:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0724b1b8]) (logid:5c91169b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-90c90027]) (logid:4ff78496) No inactive management server node found 2025-06-29 03:54:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-90c90027]) (logid:4ff78496) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:08,414 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2135a0d3]) (logid:71ac2ec3) HA health check task is running... 2025-06-29 03:54:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da22ada9]) (logid:6b8a10bd) No inactive management server node found 2025-06-29 03:54:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da22ada9]) (logid:6b8a10bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:10,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21766f82]) (logid:b6f32194) No inactive management server node found 2025-06-29 03:54:10,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21766f82]) (logid:b6f32194) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1441c54b]) (logid:f9563e7c) No inactive management server node found 2025-06-29 03:54:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1441c54b]) (logid:f9563e7c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:12,416 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d769f922]) (logid:2d233505) HA health check task is running... 2025-06-29 03:54:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f169bc1d]) (logid:71caee64) Found 0 routers to update status. 2025-06-29 03:54:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f169bc1d]) (logid:71caee64) Found 0 VPC's to update Redundant State. 2025-06-29 03:54:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f169bc1d]) (logid:71caee64) Found 0 networks to update RvR status. 2025-06-29 03:54:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71560077]) (logid:1f608280) No inactive management server node found 2025-06-29 03:54:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71560077]) (logid:1f608280) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92575526]) (logid:6454a6b8) No inactive management server node found 2025-06-29 03:54:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92575526]) (logid:6454a6b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:15,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8372: Seq 7-8372: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:54:15,276 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8372 2025-06-29 03:54:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c175b88]) (logid:b9065720) No inactive management server node found 2025-06-29 03:54:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c175b88]) (logid:b9065720) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:16,417 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e50a4fe4]) (logid:e31e2c90) HA health check task is running... 2025-06-29 03:54:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31baead4]) (logid:fc283bae) No inactive management server node found 2025-06-29 03:54:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31baead4]) (logid:fc283bae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d271cc99]) (logid:a1cdac1c) No inactive management server node found 2025-06-29 03:54:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d271cc99]) (logid:a1cdac1c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:20,419 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d6e60aa6]) (logid:0f3fc594) HA health check task is running... 2025-06-29 03:54:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d645adab]) (logid:7501da00) No inactive management server node found 2025-06-29 03:54:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d645adab]) (logid:7501da00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:22,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8fbf717]) (logid:263164fc) No inactive management server node found 2025-06-29 03:54:22,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d8fbf717]) (logid:263164fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ed975ae]) (logid:12062958) No inactive management server node found 2025-06-29 03:54:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ed975ae]) (logid:12062958) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:24,420 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e8f27ec3]) (logid:0cc7d231) HA health check task is running... 2025-06-29 03:54:25,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8373: Seq 7-8373: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:54:25,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8373 2025-06-29 03:54:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a0d376a]) (logid:3f74d766) No inactive management server node found 2025-06-29 03:54:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a0d376a]) (logid:3f74d766) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b46a38fb]) (logid:3299bbbc) No inactive management server node found 2025-06-29 03:54:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b46a38fb]) (logid:3299bbbc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5a51dbf4]) (logid:fa08e9f4) Enabled non-edge zones available for scan: 1 2025-06-29 03:54:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5a51dbf4]) (logid:fa08e9f4) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:54:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5a51dbf4]) (logid:fa08e9f4) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:54:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1817a977]) (logid:6f8b9518) Enabled non-edge zones available for scan: 1 2025-06-29 03:54:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1817a977]) (logid:6f8b9518) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:54:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5325d582]) (logid:a27b839a) No inactive management server node found 2025-06-29 03:54:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5325d582]) (logid:a27b839a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:28,422 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6782dd30]) (logid:d91e78d4) HA health check task is running... 2025-06-29 03:54:29,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06a1b612]) (logid:a285f103) No inactive management server node found 2025-06-29 03:54:29,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06a1b612]) (logid:a285f103) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:31,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e5714a6]) (logid:3d2deb2d) No inactive management server node found 2025-06-29 03:54:31,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e5714a6]) (logid:3d2deb2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:32,423 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-35ac9032]) (logid:ccdc35af) HA health check task is running... 2025-06-29 03:54:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a275ed48]) (logid:34d6ded4) No inactive management server node found 2025-06-29 03:54:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a275ed48]) (logid:34d6ded4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:34,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4ca206d]) (logid:464078ed) No inactive management server node found 2025-06-29 03:54:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4ca206d]) (logid:464078ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:35,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8374: Seq 7-8374: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:54:35,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8374 2025-06-29 03:54:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b35818c]) (logid:f11f0947) No inactive management server node found 2025-06-29 03:54:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b35818c]) (logid:f11f0947) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:36,424 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-47c68d24]) (logid:e3651116) HA health check task is running... 2025-06-29 03:54:37,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74f28eb0]) (logid:b253104b) No inactive management server node found 2025-06-29 03:54:37,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74f28eb0]) (logid:b253104b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:38,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a3bcd96]) (logid:2e8bc484) No inactive management server node found 2025-06-29 03:54:38,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a3bcd96]) (logid:2e8bc484) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:40,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d3ce7bb]) (logid:92535bd9) No inactive management server node found 2025-06-29 03:54:40,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d3ce7bb]) (logid:92535bd9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:40,426 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ea087d67]) (logid:afb7bdad) HA health check task is running... 2025-06-29 03:54:41,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8b4b997]) (logid:9fac6a9d) No inactive management server node found 2025-06-29 03:54:41,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8b4b997]) (logid:9fac6a9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-71bdcf1c]) (logid:e5d456fb) DbCollector is running... 2025-06-29 03:54:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cbfb2fbb]) (logid:221f7bf1) Found 0 routers to update status. 2025-06-29 03:54:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cbfb2fbb]) (logid:221f7bf1) Found 0 VPC's to update Redundant State. 2025-06-29 03:54:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cbfb2fbb]) (logid:221f7bf1) Found 0 networks to update RvR status. 2025-06-29 03:54:43,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-424ccb4f]) (logid:ac17431c) No inactive management server node found 2025-06-29 03:54:43,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-424ccb4f]) (logid:ac17431c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) ManagementServerCollector is running... 2025-06-29 03:54:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) DB properties were already loaded 2025-06-29 03:54:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:54:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Execution of process [4065577] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:54:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Exit value of process [4065577] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:54:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Process [4065577] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:54:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) usage install: null 2025-06-29 03:54:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Sessions active: 0 2025-06-29 03:54:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:54:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065580] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:54:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:54:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065585] for command [/bin/bash -c uname -r ]. 2025-06-29 03:54:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:54:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065586] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:54:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) system memory from /proc: 4887515136 2025-06-29 03:54:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:54:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065591] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:54:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) free memory from /proc: 187629568 2025-06-29 03:54:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:54:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065596] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:54:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) used memory from /proc: 1073260 2025-06-29 03:54:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:54:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065597] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:54:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:54:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065598] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:54:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:54:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065599] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:54:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:54:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065606] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:54:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:54:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Successfully executed process [4065607] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:54:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail buffersdirect.capacity, 242727 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail buffersdirect.count, 75 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail buffersdirect.used, 242727 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail buffersmapped.capacity, 0 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail buffersmapped.count, 0 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail buffersmapped.used, 0 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail gcPS-Scavenge.count, 105 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail gcPS-Scavenge.time, 486 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail jvmuptime, 1143830 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memoryheap.committed, 545259520 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memoryheap.init, 77594624 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memoryheap.usage, 0.2684024558293888 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorynon-heap.committed, 191234048 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorynon-heap.max, -1 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorynon-heap.usage, -1.86902056E8 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorynon-heap.used, 186902056 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09688562216742203 2025-06-29 03:54:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.1936391925217449 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.Compressed-Class-Space.usage, 0.012212499976158142 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.Metaspace.usage, 0.993743502803243 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.PS-Eden-Space.usage, 0.49832464031021356 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.PS-Old-Gen.usage, 0.11202547009809251 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorypools.PS-Survivor-Space.usage, 0.9976715543376866 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorytotal.committed, 736493568 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorytotal.init, 85262336 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorytotal.max, 1908932607 2025-06-29 03:54:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail memorytotal.used, 699264256 2025-06-29 03:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail threadsdeadlocks, [] 2025-06-29 03:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail threadsnew.count, 0 2025-06-29 03:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:54:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-8d203884]) (logid:fb6f96d4) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":33543416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.69,"totalInit":85262336,"totalUsed":698613696,"totalCommitted":736493568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":512362200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":187629568,"systemMemoryUsed":1073260,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.69,0.36,0.24],"systemCyclesUsage":[66401475,53825362,4589058331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:54:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 37, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":33543416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.69,"totalInit":85262336,"totalUsed":698613696,"totalCommitted":736493568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":512362200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":187629568,"systemMemoryUsed":1073260,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.69,0.36,0.24],"systemCyclesUsage":[66401475,53825362,4589058331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:54:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:54:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-11:[ctx-37cdcb51]) (logid:c27d2a3d) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":33543416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.69,"totalInit":85262336,"totalUsed":698613696,"totalCommitted":736493568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":512362200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":187629568,"systemMemoryUsed":1073260,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.69,0.36,0.24],"systemCyclesUsage":[66401475,53825362,4589058331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:54:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 03:54:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 37, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":545259520,"freeJvmMemoryBytes":33543416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.69,"totalInit":85262336,"totalUsed":698613696,"totalCommitted":736493568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":512362200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":187629568,"systemMemoryUsed":1073260,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.69,0.36,0.24],"systemCyclesUsage":[66401475,53825362,4589058331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:54:44,427 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-62463533]) (logid:271b631a) HA health check task is running... 2025-06-29 03:54:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-192f5b86]) (logid:36901601) No inactive management server node found 2025-06-29 03:54:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-192f5b86]) (logid:36901601) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:45,256 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8375: Seq 7-8375: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:54:45,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8375 2025-06-29 03:54:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:54:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:54:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-063da3c5]) (logid:04e9c698) No inactive management server node found 2025-06-29 03:54:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-063da3c5]) (logid:04e9c698) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:54:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:54:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:54:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:54:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:54:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:54:47,075 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:54:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:54:47,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0cc73a0]) (logid:7103401a) No inactive management server node found 2025-06-29 03:54:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0cc73a0]) (logid:7103401a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a6fc2673]) (logid:097af806) ClusterDRS.poll is being called at 2025-06-29T01:55:00+0000 2025-06-29 03:54:48,073 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a6fc2673]) (logid:097af806) Removed 0 old drs migration plans 2025-06-29 03:54:48,087 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-42216dff]) (logid:9be78f46) Resetting hosts suitable for reconnect 2025-06-29 03:54:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-42216dff]) (logid:9be78f46) Completed resetting hosts suitable for reconnect 2025-06-29 03:54:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-42216dff]) (logid:9be78f46) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:54:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-42216dff]) (logid:9be78f46) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:54:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-42216dff]) (logid:9be78f46) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:54:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-42216dff]) (logid:9be78f46) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:54:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-fa2fc37d]) (logid:5d4b4bb6) VM scheduler.poll is being called at 2025-06-29T01:55:00+0000 2025-06-29 03:54:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-fa2fc37d]) (logid:5d4b4bb6) Got 0 scheduled jobs to be executed at 2025-06-29T01:55:00+0000 2025-06-29 03:54:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-fa2fc37d]) (logid:5d4b4bb6) Cleaned up 0 VM scheduled job entries 2025-06-29 03:54:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-35ef1f19]) (logid:b1a3d3fc) VM Operation Thread Running 2025-06-29 03:54:48,428 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bbf539cf]) (logid:6e48433f) HA health check task is running... 2025-06-29 03:54:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b924dd38]) (logid:8e6249be) No inactive management server node found 2025-06-29 03:54:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b924dd38]) (logid:8e6249be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-896398b0]) (logid:ea3300fc) No inactive management server node found 2025-06-29 03:54:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-896398b0]) (logid:ea3300fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:52,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05b34bf3]) (logid:2636045b) No inactive management server node found 2025-06-29 03:54:52,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05b34bf3]) (logid:2636045b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:52,429 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-308521fd]) (logid:f42735cb) HA health check task is running... 2025-06-29 03:54:53,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62958f9e]) (logid:d5890d33) No inactive management server node found 2025-06-29 03:54:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62958f9e]) (logid:d5890d33) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:55,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8377: Seq 7-8377: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:54:55,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8377 2025-06-29 03:54:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e213fcd6]) (logid:276ccd6e) No inactive management server node found 2025-06-29 03:54:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e213fcd6]) (logid:276ccd6e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:56,431 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ec93bd06]) (logid:6e528dba) HA health check task is running... 2025-06-29 03:54:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbef8f5c]) (logid:51b06401) No inactive management server node found 2025-06-29 03:54:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbef8f5c]) (logid:51b06401) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-54186e22]) (logid:3db4bef5) Enabled non-edge zones available for scan: 1 2025-06-29 03:54:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-54186e22]) (logid:3db4bef5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:54:58,094 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-2f9b2518]) (logid:7a8f0b7e) AutoScaling Monitor is running... 2025-06-29 03:54:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-54186e22]) (logid:3db4bef5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:54:58,098 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-6f41c996]) (logid:ab7eefce) Starting to remove all volume_stats rows older than [Sat Jun 28 15:54:58 CEST 2025]. 2025-06-29 03:54:58,099 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-6f41c996]) (logid:ab7eefce) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:54:58 CEST 2025]. 2025-06-29 03:54:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-70e97763]) (logid:84b34987) Enabled non-edge zones available for scan: 1 2025-06-29 03:54:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-70e97763]) (logid:84b34987) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:54:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c3491de]) (logid:16f1ffdf) No inactive management server node found 2025-06-29 03:54:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c3491de]) (logid:16f1ffdf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:54:58,603 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-10148f96]) (logid:75ddec72) Starting to remove all vm_stats rows older than [Sat Jun 28 15:54:58 CEST 2025]. 2025-06-29 03:54:58,636 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-10148f96]) (logid:75ddec72) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:54:58 CEST 2025]. 2025-06-29 03:54:59,527 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:54:59,529 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:54:59,530 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) Seq 2-4725683383995203666: Routed from 90520738888109 2025-06-29 03:54:59,560 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:54:59,561 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 03:54:59,603 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) Seq 1-4725683383995203666: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:54:59,604 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:54:59,605 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) Seq 2-3926857400090361948: Routed from 90520738888109 2025-06-29 03:54:59,650 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-c39e49b1]) (logid:7b870aa6) Seq 2-3926857400090361948: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:54:59,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6e814c4]) (logid:23e73649) No inactive management server node found 2025-06-29 03:54:59,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6e814c4]) (logid:23e73649) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:00,432 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-13aa4996]) (logid:62d702c0) HA health check task is running... 2025-06-29 03:55:01,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7633491]) (logid:934e5f4b) No inactive management server node found 2025-06-29 03:55:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7633491]) (logid:934e5f4b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:02,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98c73819]) (logid:8a59ecf5) No inactive management server node found 2025-06-29 03:55:02,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98c73819]) (logid:8a59ecf5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:03,010 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:55:03,014 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:55:03,015 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) Seq 2-4725683383995203667: Routed from 90520738888109 2025-06-29 03:55:03,075 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) Seq 1-4725683383995203667: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:55:03,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:55:03,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) Seq 2-3926857400090361949: Routed from 90520738888109 2025-06-29 03:55:03,210 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-1ddccc15]) (logid:d8e9f018) Seq 2-3926857400090361949: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:55:03,581 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) StorageCollector is running... 2025-06-29 03:55:03,586 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:55:03,586 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:55:03,586 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:55:03,587 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-3938116399158788122: Routed from 90520738888109 2025-06-29 03:55:03,642 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 4-3938116399158788122: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:55:03,646 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:55:03,646 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:55:03,646 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:55:03,647 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-4725683383995203668: Routed from 90520738888109 2025-06-29 03:55:03,694 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 1-4725683383995203668: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:55:03,705 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:55:03,705 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:55:03,705 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:55:03,706 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-3926857400090361950: Routed from 90520738888109 2025-06-29 03:55:03,717 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-3926857400090361950: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:55:03,727 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:55:03,727 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:55:03,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:55:03,728 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-4725683383995203669: Routed from 90520738888109 2025-06-29 03:55:03,814 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 1-4725683383995203669: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:55:03,815 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:55:03,815 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:55:03,816 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:55:03,817 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-3926857400090361951: Routed from 90520738888109 2025-06-29 03:55:03,863 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-aec0511f]) (logid:58b8dbb8) Seq 2-3926857400090361951: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:55:04,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-745d098a]) (logid:203414e1) No inactive management server node found 2025-06-29 03:55:04,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-745d098a]) (logid:203414e1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:04,433 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7ce2c185]) (logid:177e4cf1) HA health check task is running... 2025-06-29 03:55:05,262 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8378: Seq 7-8378: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:55:05,284 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8378 2025-06-29 03:55:05,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ed67223]) (logid:fa72e7c2) No inactive management server node found 2025-06-29 03:55:05,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ed67223]) (logid:fa72e7c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:07,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0e442ad]) (logid:28d09e1d) No inactive management server node found 2025-06-29 03:55:07,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0e442ad]) (logid:28d09e1d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:08,434 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f5fb1ade]) (logid:3652ee46) HA health check task is running... 2025-06-29 03:55:08,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e234d60b]) (logid:39e781ba) No inactive management server node found 2025-06-29 03:55:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e234d60b]) (logid:39e781ba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c61a83b]) (logid:60c9053a) No inactive management server node found 2025-06-29 03:55:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c61a83b]) (logid:60c9053a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-70511cfb]) (logid:602da0fe) No inactive management server node found 2025-06-29 03:55:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-70511cfb]) (logid:602da0fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:12,436 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-58a5a7e4]) (logid:b0cb6a9a) HA health check task is running... 2025-06-29 03:55:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cc5b03d9]) (logid:6f2d5500) Found 0 routers to update status. 2025-06-29 03:55:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cc5b03d9]) (logid:6f2d5500) Found 0 VPC's to update Redundant State. 2025-06-29 03:55:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cc5b03d9]) (logid:6f2d5500) Found 0 networks to update RvR status. 2025-06-29 03:55:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5efc92de]) (logid:ff3f540d) No inactive management server node found 2025-06-29 03:55:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5efc92de]) (logid:ff3f540d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07e9546a]) (logid:4ab5101f) No inactive management server node found 2025-06-29 03:55:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07e9546a]) (logid:4ab5101f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:15,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8379: Seq 7-8379: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:55:15,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8379 2025-06-29 03:55:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04d5a601]) (logid:d3fe5ac1) No inactive management server node found 2025-06-29 03:55:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04d5a601]) (logid:d3fe5ac1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:16,437 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-179509b5]) (logid:013967df) HA health check task is running... 2025-06-29 03:55:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fc6872f]) (logid:866674af) No inactive management server node found 2025-06-29 03:55:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fc6872f]) (logid:866674af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa3ff7b7]) (logid:dd6ed618) No inactive management server node found 2025-06-29 03:55:19,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa3ff7b7]) (logid:dd6ed618) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:20,439 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-20b57240]) (logid:e5b0404c) HA health check task is running... 2025-06-29 03:55:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2fd32d91]) (logid:6ea619cd) No inactive management server node found 2025-06-29 03:55:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2fd32d91]) (logid:6ea619cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:22,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-545898d9]) (logid:2997a7d9) No inactive management server node found 2025-06-29 03:55:22,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-545898d9]) (logid:2997a7d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50370eb3]) (logid:435b4c09) No inactive management server node found 2025-06-29 03:55:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50370eb3]) (logid:435b4c09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:24,440 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a4deb475]) (logid:d1f37af2) HA health check task is running... 2025-06-29 03:55:25,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8380: Seq 7-8380: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:55:25,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8380 2025-06-29 03:55:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-071ff929]) (logid:3d8b2536) No inactive management server node found 2025-06-29 03:55:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-071ff929]) (logid:3d8b2536) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ae26cb5]) (logid:8c40c88f) No inactive management server node found 2025-06-29 03:55:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ae26cb5]) (logid:8c40c88f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6dda5061]) (logid:d6d29dac) Enabled non-edge zones available for scan: 1 2025-06-29 03:55:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6dda5061]) (logid:d6d29dac) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:55:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6dda5061]) (logid:d6d29dac) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:55:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-97cc40c4]) (logid:5fbab173) Enabled non-edge zones available for scan: 1 2025-06-29 03:55:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-97cc40c4]) (logid:5fbab173) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:55:28,374 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7e94dc5]) (logid:c892c634) No inactive management server node found 2025-06-29 03:55:28,374 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7e94dc5]) (logid:c892c634) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:28,441 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a57e3bd0]) (logid:94a4941d) HA health check task is running... 2025-06-29 03:55:29,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-094fe0bb]) (logid:c77940d1) No inactive management server node found 2025-06-29 03:55:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-094fe0bb]) (logid:c77940d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:30,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8381: Seq 7-8381: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:55:30,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8381 2025-06-29 03:55:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa94ea75]) (logid:097a8023) No inactive management server node found 2025-06-29 03:55:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa94ea75]) (logid:097a8023) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:32,443 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7c14e9f7]) (logid:7cd1e01e) HA health check task is running... 2025-06-29 03:55:32,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a550928d]) (logid:ff3990f7) No inactive management server node found 2025-06-29 03:55:32,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a550928d]) (logid:ff3990f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25cf500d]) (logid:d6a534e8) No inactive management server node found 2025-06-29 03:55:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25cf500d]) (logid:d6a534e8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7400fd51]) (logid:2fd375e8) No inactive management server node found 2025-06-29 03:55:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7400fd51]) (logid:2fd375e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:36,444 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-bad43001]) (logid:c5897400) HA health check task is running... 2025-06-29 03:55:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-875caba7]) (logid:46e8ea48) No inactive management server node found 2025-06-29 03:55:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-875caba7]) (logid:46e8ea48) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e2aaabc]) (logid:361123ac) No inactive management server node found 2025-06-29 03:55:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e2aaabc]) (logid:361123ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:40,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8382: Seq 7-8382: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:55:40,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8382 2025-06-29 03:55:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a10efb1]) (logid:54ac8fdf) No inactive management server node found 2025-06-29 03:55:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a10efb1]) (logid:54ac8fdf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:40,446 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d25d17a6]) (logid:c99caf5e) HA health check task is running... 2025-06-29 03:55:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ef4a3b2]) (logid:af611769) No inactive management server node found 2025-06-29 03:55:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ef4a3b2]) (logid:af611769) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-ab07d5f2]) (logid:d4f82b23) DbCollector is running... 2025-06-29 03:55:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3b7efae6]) (logid:881b4c2d) Found 0 routers to update status. 2025-06-29 03:55:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3b7efae6]) (logid:881b4c2d) Found 0 VPC's to update Redundant State. 2025-06-29 03:55:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3b7efae6]) (logid:881b4c2d) Found 0 networks to update RvR status. 2025-06-29 03:55:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32387625]) (logid:640b4e9c) No inactive management server node found 2025-06-29 03:55:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32387625]) (logid:640b4e9c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) ManagementServerCollector is running... 2025-06-29 03:55:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) DB properties were already loaded 2025-06-29 03:55:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:55:44,057 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Execution of process [4065614] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:55:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Exit value of process [4065614] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:55:44,057 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Process [4065614] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:55:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) usage install: null 2025-06-29 03:55:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Sessions active: 0 2025-06-29 03:55:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:55:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065617] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:55:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:55:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065622] for command [/bin/bash -c uname -r ]. 2025-06-29 03:55:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:55:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065623] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:55:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) system memory from /proc: 4887515136 2025-06-29 03:55:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:55:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065628] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:55:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) free memory from /proc: 196911104 2025-06-29 03:55:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:55:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065633] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:55:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) used memory from /proc: 1039452 2025-06-29 03:55:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:55:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065634] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:55:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:55:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065635] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:55:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:55:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065636] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:55:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:55:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065643] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:55:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:55:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Successfully executed process [4065644] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:55:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail buffersdirect.capacity, 242727 2025-06-29 03:55:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail buffersdirect.count, 75 2025-06-29 03:55:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail buffersdirect.used, 242727 2025-06-29 03:55:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail buffersmapped.capacity, 0 2025-06-29 03:55:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail buffersmapped.count, 0 2025-06-29 03:55:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail buffersmapped.used, 0 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail gcPS-Scavenge.count, 106 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail gcPS-Scavenge.time, 514 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail jvmuptime, 1203831 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memoryheap.committed, 536870912 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memoryheap.init, 77594624 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memoryheap.usage, 0.1195858497273886 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorynon-heap.committed, 191299584 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorynon-heap.max, -1 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorynon-heap.usage, -1.87054896E8 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorynon-heap.used, 187054896 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.0972271977472674 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19435985769987002 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.Compressed-Class-Space.usage, 0.012212499976158142 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.Metaspace.usage, 0.9934312880551576 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.PS-Eden-Space.usage, 0.042440270734721945 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 03:55:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 03:55:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorytotal.committed, 728170496 2025-06-29 03:55:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorytotal.init, 85262336 2025-06-29 03:55:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorytotal.max, 1908932607 2025-06-29 03:55:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail memorytotal.used, 415336224 2025-06-29 03:55:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail threadsdeadlocks, [] 2025-06-29 03:55:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail threadsnew.count, 0 2025-06-29 03:55:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:55:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-294a5186]) (logid:2913cad2) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":309219088,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.52,"totalInit":85262336,"totalUsed":414703928,"totalCommitted":728170496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228281328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":196911104,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.52,0.38,0.26],"systemCyclesUsage":[66402091,53825685,4589081135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:55:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 39, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":309219088,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.52,"totalInit":85262336,"totalUsed":414703928,"totalCommitted":728170496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228281328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":196911104,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.52,0.38,0.26],"systemCyclesUsage":[66402091,53825685,4589081135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:55:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:55:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:55:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-12:[ctx-30b21e16]) (logid:6f765055) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":309219088,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.52,"totalInit":85262336,"totalUsed":414703928,"totalCommitted":728170496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228281328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":196911104,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.52,0.38,0.26],"systemCyclesUsage":[66402091,53825685,4589081135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:55:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 39, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":309219088,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.52,"totalInit":85262336,"totalUsed":414703928,"totalCommitted":728170496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228281328,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":196911104,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.52,0.38,0.26],"systemCyclesUsage":[66402091,53825685,4589081135],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:55:44,447 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-6bec7ba1]) (logid:833f7365) HA health check task is running... 2025-06-29 03:55:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b6a9077]) (logid:69d044f5) No inactive management server node found 2025-06-29 03:55:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b6a9077]) (logid:69d044f5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:55:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:55:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d44d9c]) (logid:09ada7e3) No inactive management server node found 2025-06-29 03:55:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d44d9c]) (logid:09ada7e3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:55:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:55:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:55:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:55:47,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:55:47,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:55:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:55:47,083 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:55:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26a59b88]) (logid:f3fa8ce3) No inactive management server node found 2025-06-29 03:55:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26a59b88]) (logid:f3fa8ce3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e34f9fc3]) (logid:4d5f1a4d) ClusterDRS.poll is being called at 2025-06-29T01:56:00+0000 2025-06-29 03:55:48,073 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e34f9fc3]) (logid:4d5f1a4d) Removed 0 old drs migration plans 2025-06-29 03:55:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-97b7eaf4]) (logid:4a8e3a04) VM scheduler.poll is being called at 2025-06-29T01:56:00+0000 2025-06-29 03:55:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-97b7eaf4]) (logid:4a8e3a04) Got 0 scheduled jobs to be executed at 2025-06-29T01:56:00+0000 2025-06-29 03:55:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-97b7eaf4]) (logid:4a8e3a04) Cleaned up 0 VM scheduled job entries 2025-06-29 03:55:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-4280b176]) (logid:666bd6ae) VM Operation Thread Running 2025-06-29 03:55:48,448 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-9e052416]) (logid:06879686) HA health check task is running... 2025-06-29 03:55:49,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a19d4fb8]) (logid:7edc5a3a) No inactive management server node found 2025-06-29 03:55:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a19d4fb8]) (logid:7edc5a3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:50,257 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8384: Seq 7-8384: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:55:50,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8384 2025-06-29 03:55:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e8b0a7f]) (logid:dc46154f) No inactive management server node found 2025-06-29 03:55:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e8b0a7f]) (logid:dc46154f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1465245]) (logid:8a0b4d0c) No inactive management server node found 2025-06-29 03:55:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1465245]) (logid:8a0b4d0c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:52,450 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-da3ea63c]) (logid:ab14975c) HA health check task is running... 2025-06-29 03:55:53,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e969bab]) (logid:1c1064ab) No inactive management server node found 2025-06-29 03:55:53,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e969bab]) (logid:1c1064ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:55,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4888f969]) (logid:59acc216) No inactive management server node found 2025-06-29 03:55:55,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4888f969]) (logid:59acc216) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:56,451 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b25be250]) (logid:469afcbf) HA health check task is running... 2025-06-29 03:55:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-578b5ad6]) (logid:be89bfbe) No inactive management server node found 2025-06-29 03:55:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-578b5ad6]) (logid:be89bfbe) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c8415ca]) (logid:ad482a26) Enabled non-edge zones available for scan: 1 2025-06-29 03:55:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0c8415ca]) (logid:ad482a26) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:55:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c8415ca]) (logid:ad482a26) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:55:58,096 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-651606f6]) (logid:18bab26c) AutoScaling Monitor is running... 2025-06-29 03:55:58,101 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-42cc2236]) (logid:10ca1d15) Starting to remove all volume_stats rows older than [Sat Jun 28 15:55:58 CEST 2025]. 2025-06-29 03:55:58,102 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-42cc2236]) (logid:10ca1d15) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:55:58 CEST 2025]. 2025-06-29 03:55:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2fd5b445]) (logid:ef960928) Enabled non-edge zones available for scan: 1 2025-06-29 03:55:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2fd5b445]) (logid:ef960928) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:55:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45ef4e66]) (logid:0e525c65) No inactive management server node found 2025-06-29 03:55:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45ef4e66]) (logid:0e525c65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:55:58,638 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-1f1400ea]) (logid:28f00a03) Starting to remove all vm_stats rows older than [Sat Jun 28 15:55:58 CEST 2025]. 2025-06-29 03:55:58,661 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-1f1400ea]) (logid:28f00a03) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:55:58 CEST 2025]. 2025-06-29 03:55:59,653 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:55:59,655 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:55:59,656 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) Seq 2-4725683383995203670: Routed from 90520738888109 2025-06-29 03:55:59,700 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) Seq 1-4725683383995203670: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:55:59,701 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:55:59,702 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) Seq 2-3926857400090361952: Routed from 90520738888109 2025-06-29 03:55:59,747 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-26fa32e4]) (logid:08f1c462) Seq 2-3926857400090361952: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:55:59,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-670702c3]) (logid:908e932a) No inactive management server node found 2025-06-29 03:55:59,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-670702c3]) (logid:908e932a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:00,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8385: Seq 7-8385: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:00,280 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8385 2025-06-29 03:56:00,452 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7ae1b8d2]) (logid:cdf9117f) HA health check task is running... 2025-06-29 03:56:01,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0edf07d1]) (logid:24ea04e8) No inactive management server node found 2025-06-29 03:56:01,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0edf07d1]) (logid:24ea04e8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:02,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb047ba]) (logid:2611a6dc) No inactive management server node found 2025-06-29 03:56:02,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb047ba]) (logid:2611a6dc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:56:03,268 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:56:03,271 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:56:03,272 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) Seq 2-4725683383995203671: Routed from 90520738888109 2025-06-29 03:56:03,328 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) Seq 1-4725683383995203671: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:56:03,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:56:03,377 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) Seq 2-3926857400090361953: Routed from 90520738888109 2025-06-29 03:56:03,466 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7daa7f8a]) (logid:ae6f9093) Seq 2-3926857400090361953: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:56:03,864 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) StorageCollector is running... 2025-06-29 03:56:03,869 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:56:03,869 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:56:03,869 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:56:03,870 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 2-3938116399158788123: Routed from 90520738888109 2025-06-29 03:56:03,924 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 4-3938116399158788123: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:56:03,928 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:56:03,928 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:56:03,928 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:56:03,929 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 2-4725683383995203672: Routed from 90520738888109 2025-06-29 03:56:03,979 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 1-4725683383995203672: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:56:04,006 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:56:04,006 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:56:04,007 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:56:04,009 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 2-4725683383995203673: Routed from 90520738888109 2025-06-29 03:56:04,100 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 1-4725683383995203673: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:56:04,110 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:56:04,110 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:56:04,110 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:56:04,111 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 2-4725683383995203674: Routed from 90520738888109 2025-06-29 03:56:04,196 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 1-4725683383995203674: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:56:04,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:56:04,197 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:56:04,197 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:56:04,198 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 2-3926857400090361954: Routed from 90520738888109 2025-06-29 03:56:04,203 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-43cc36be]) (logid:3f6a7652) Seq 2-3926857400090361954: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:56:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e638c65]) (logid:1fe09e1c) No inactive management server node found 2025-06-29 03:56:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e638c65]) (logid:1fe09e1c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:04,453 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c1aeb303]) (logid:f078bfa8) HA health check task is running... 2025-06-29 03:56:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8660d6b]) (logid:7fc886e2) No inactive management server node found 2025-06-29 03:56:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8660d6b]) (logid:7fc886e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-996a3825]) (logid:d064c90a) No inactive management server node found 2025-06-29 03:56:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-996a3825]) (logid:d064c90a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:08,454 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f03a118b]) (logid:b29f4c06) HA health check task is running... 2025-06-29 03:56:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e6bd884]) (logid:c6d8190d) No inactive management server node found 2025-06-29 03:56:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e6bd884]) (logid:c6d8190d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:10,258 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8386: Seq 7-8386: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:10,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8386 2025-06-29 03:56:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b02c66e7]) (logid:be59bb13) No inactive management server node found 2025-06-29 03:56:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b02c66e7]) (logid:be59bb13) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b35104d]) (logid:548c0591) No inactive management server node found 2025-06-29 03:56:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b35104d]) (logid:548c0591) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:12,456 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-bf5ada05]) (logid:13fae06d) HA health check task is running... 2025-06-29 03:56:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad8409ee]) (logid:2c7b3c58) Found 0 routers to update status. 2025-06-29 03:56:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad8409ee]) (logid:2c7b3c58) Found 0 VPC's to update Redundant State. 2025-06-29 03:56:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad8409ee]) (logid:2c7b3c58) Found 0 networks to update RvR status. 2025-06-29 03:56:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b474cc49]) (logid:9dbd0720) No inactive management server node found 2025-06-29 03:56:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b474cc49]) (logid:9dbd0720) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daebfbf7]) (logid:27949111) No inactive management server node found 2025-06-29 03:56:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daebfbf7]) (logid:27949111) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:15,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8387: Seq 7-8387: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:15,280 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8387 2025-06-29 03:56:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9effba1d]) (logid:2d2e9567) No inactive management server node found 2025-06-29 03:56:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9effba1d]) (logid:2d2e9567) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:16,457 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0627c457]) (logid:bd3e4e68) HA health check task is running... 2025-06-29 03:56:17,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5d74808]) (logid:6a9f26d2) No inactive management server node found 2025-06-29 03:56:17,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5d74808]) (logid:6a9f26d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f7139a60]) (logid:c47f81cd) Resetting hosts suitable for reconnect 2025-06-29 03:56:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f7139a60]) (logid:c47f81cd) Completed resetting hosts suitable for reconnect 2025-06-29 03:56:18,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f7139a60]) (logid:c47f81cd) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:56:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f7139a60]) (logid:c47f81cd) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:56:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f7139a60]) (logid:c47f81cd) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:56:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-f7139a60]) (logid:c47f81cd) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:56:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46c6bc7f]) (logid:e94aab30) No inactive management server node found 2025-06-29 03:56:19,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46c6bc7f]) (logid:e94aab30) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:20,458 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a4c0bfb3]) (logid:2159b4ce) HA health check task is running... 2025-06-29 03:56:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72f51541]) (logid:cdcd3f93) No inactive management server node found 2025-06-29 03:56:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72f51541]) (logid:cdcd3f93) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c643fef7]) (logid:2e0363b5) No inactive management server node found 2025-06-29 03:56:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c643fef7]) (logid:2e0363b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-233cc7e8]) (logid:b02a162f) No inactive management server node found 2025-06-29 03:56:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-233cc7e8]) (logid:b02a162f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:24,460 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-25fe8da4]) (logid:7b93dadf) HA health check task is running... 2025-06-29 03:56:25,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8388: Seq 7-8388: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:25,280 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8388 2025-06-29 03:56:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6105685e]) (logid:19175b8a) No inactive management server node found 2025-06-29 03:56:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6105685e]) (logid:19175b8a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd22b3ed]) (logid:f02bf47e) No inactive management server node found 2025-06-29 03:56:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd22b3ed]) (logid:f02bf47e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-04b7b286]) (logid:af7b6082) Enabled non-edge zones available for scan: 1 2025-06-29 03:56:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-04b7b286]) (logid:af7b6082) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:56:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-04b7b286]) (logid:af7b6082) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:56:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-25f8e98a]) (logid:fc66d409) Enabled non-edge zones available for scan: 1 2025-06-29 03:56:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-25f8e98a]) (logid:fc66d409) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:56:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fa33c53]) (logid:5411c62d) No inactive management server node found 2025-06-29 03:56:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fa33c53]) (logid:5411c62d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:28,461 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-18974e79]) (logid:a53ec1fd) HA health check task is running... 2025-06-29 03:56:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-163b691d]) (logid:eb51909d) No inactive management server node found 2025-06-29 03:56:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-163b691d]) (logid:eb51909d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab623d9a]) (logid:2cbd9560) No inactive management server node found 2025-06-29 03:56:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab623d9a]) (logid:2cbd9560) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:32,462 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ea457741]) (logid:15d6822c) HA health check task is running... 2025-06-29 03:56:32,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cdc7bb4]) (logid:878128b0) No inactive management server node found 2025-06-29 03:56:32,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cdc7bb4]) (logid:878128b0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84bf4a25]) (logid:afbacc3a) No inactive management server node found 2025-06-29 03:56:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84bf4a25]) (logid:afbacc3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:35,259 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8389: Seq 7-8389: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:35,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8389 2025-06-29 03:56:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cad0c1fa]) (logid:88ac9ee1) No inactive management server node found 2025-06-29 03:56:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cad0c1fa]) (logid:88ac9ee1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:36,464 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8143a35e]) (logid:31f95c75) HA health check task is running... 2025-06-29 03:56:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6531f7e]) (logid:f22cc8f1) No inactive management server node found 2025-06-29 03:56:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6531f7e]) (logid:f22cc8f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19aeb605]) (logid:27d74101) No inactive management server node found 2025-06-29 03:56:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19aeb605]) (logid:27d74101) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:40,260 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8390: Seq 7-8390: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:40,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8390 2025-06-29 03:56:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b030c2]) (logid:6ad23878) No inactive management server node found 2025-06-29 03:56:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b030c2]) (logid:6ad23878) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:40,465 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-47263e2d]) (logid:e001e71b) HA health check task is running... 2025-06-29 03:56:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6130d48a]) (logid:e0287563) No inactive management server node found 2025-06-29 03:56:41,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6130d48a]) (logid:e0287563) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:43,007 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-6c08a87a]) (logid:74a4f607) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 03:56:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-9f6eac41]) (logid:b341b455) DbCollector is running... 2025-06-29 03:56:43,093 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-671a4fbc]) (logid:ece4b1bf) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 03:56:43,094 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-671a4fbc]) (logid:ece4b1bf) Snapshot scheduler is being called at [2025-06-29T01:56:43+0000]. 2025-06-29 03:56:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-671a4fbc]) (logid:ece4b1bf) There are [0] scheduled snapshots to be executed at [2025-06-29T01:56:43+0000]. 2025-06-29 03:56:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-efa1c279]) (logid:840a3a6c) Found 0 running routers. 2025-06-29 03:56:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4808cf8f]) (logid:8c613ad8) Found 0 routers to update status. 2025-06-29 03:56:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4808cf8f]) (logid:8c613ad8) Found 0 VPC's to update Redundant State. 2025-06-29 03:56:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4808cf8f]) (logid:8c613ad8) Found 0 networks to update RvR status. 2025-06-29 03:56:43,109 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Found 1 running routers. 2025-06-29 03:56:43,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:56:43,117 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Seq 2-3926857400090361955: Routed from 90520738888109 2025-06-29 03:56:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-43888307]) (logid:53bb3589) External devices are not used. Skipping external device usage collection 2025-06-29 03:56:43,117 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Seq 2-3926857400090361955: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:43,126 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-b2dbe45f]) (logid:dcff315f) Started resource counters recalculation periodic task. 2025-06-29 03:56:43,129 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-b2dbe45f]) (logid:dcff315f) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 03:56:43,304 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-0a4fe0eb]) (logid:b2c3313e) Backup backup.poll is being called at 2025-06-29T01:56:43+0000 2025-06-29 03:56:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2d9cfe7]) (logid:9d9b6062) No inactive management server node found 2025-06-29 03:56:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2d9cfe7]) (logid:9d9b6062) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:56:43,585 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-b2dbe45f]) (logid:dcff315f) Finished resource counters recalculation periodic task. 2025-06-29 03:56:43,696 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 2-3926857400090361955: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:43,696 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Seq 2-3926857400090361955: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:56:43,696 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 03:56:43,700 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 03:56:43,748 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 03:56:43,749 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Seq 2-3926857400090361956: Routed from 90520738888109 2025-06-29 03:56:43,750 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Seq 2-3926857400090361956: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) ManagementServerCollector is running... 2025-06-29 03:56:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) DB properties were already loaded 2025-06-29 03:56:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:56:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Execution of process [4065659] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:56:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Exit value of process [4065659] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:56:44,058 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Process [4065659] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:56:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) usage install: null 2025-06-29 03:56:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Sessions active: 0 2025-06-29 03:56:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:56:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065662] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:56:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:56:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065667] for command [/bin/bash -c uname -r ]. 2025-06-29 03:56:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:56:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065668] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:56:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) system memory from /proc: 4887515136 2025-06-29 03:56:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:56:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065673] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:56:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) free memory from /proc: 206995456 2025-06-29 03:56:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:56:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065678] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:56:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) used memory from /proc: 1039452 2025-06-29 03:56:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:56:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065679] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:56:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:56:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065680] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:56:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:56:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065681] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:56:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:56:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065688] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:56:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:56:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Successfully executed process [4065689] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail buffersdirect.capacity, 242727 2025-06-29 03:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail buffersdirect.count, 75 2025-06-29 03:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail buffersdirect.used, 242727 2025-06-29 03:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail buffersmapped.capacity, 0 2025-06-29 03:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail buffersmapped.count, 0 2025-06-29 03:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail buffersmapped.used, 0 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail gcPS-Scavenge.count, 106 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail gcPS-Scavenge.time, 514 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail jvmuptime, 1263834 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memoryheap.committed, 536870912 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memoryheap.init, 77594624 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memoryheap.usage, 0.14004734524394483 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorynon-heap.committed, 191430656 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorynon-heap.max, -1 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorynon-heap.usage, -1.87277376E8 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorynon-heap.used, 187277376 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30357456140350875 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.09784786556918156 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.19534192021861566 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.Compressed-Class-Space.usage, 0.01221301406621933 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.Metaspace.usage, 0.9931387161582638 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.PS-Eden-Space.usage, 0.10152060790822016 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorytotal.committed, 728301568 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorytotal.init, 85262336 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorytotal.max, 1908932607 2025-06-29 03:56:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail memorytotal.used, 454618320 2025-06-29 03:56:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail threadsdeadlocks, [] 2025-06-29 03:56:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail threadsnew.count, 0 2025-06-29 03:56:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:56:44,135 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-f5b5c461]) (logid:a0014f6a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":270141384,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1263761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":454002784,"totalCommitted":728301568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":267340944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":206995456,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.31,0.24],"systemCyclesUsage":[66402477,53826000,4589104172],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:56:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 41, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":270141384,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1263761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":454002784,"totalCommitted":728301568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":267340944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":206995456,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.31,0.24],"systemCyclesUsage":[66402477,53826000,4589104172],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:56:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:56:44,145 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-12:[ctx-c938fb4a]) (logid:29637db8) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":270141384,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1263761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":454002784,"totalCommitted":728301568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":267340944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":206995456,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.31,0.24],"systemCyclesUsage":[66402477,53826000,4589104172],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:56:44,145 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:56:44,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 41, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":270141384,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1263761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":454002784,"totalCommitted":728301568,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":267340944,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":106,"systemMemoryTotal":4887515136,"systemMemoryFree":206995456,"systemMemoryUsed":1039452,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.31,0.24],"systemCyclesUsage":[66402477,53826000,4589104172],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:56:44,467 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5c8fef81]) (logid:62d0353e) HA health check task is running... 2025-06-29 03:56:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5d6d84d]) (logid:0802905d) No inactive management server node found 2025-06-29 03:56:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5d6d84d]) (logid:0802905d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:45,137 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 2-3926857400090361956: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.7d882f26-c8b7-488b-8d49-ecfaf9ddcf9e","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:45,138 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-efa15e83]) (logid:1f0bd786) Seq 2-3926857400090361956: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 03:56:45,140 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 03:56:45,143 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 03:56:45,144 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Seq 2-3926857400090361957: Routed from 90520738888109 2025-06-29 03:56:45,144 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Seq 2-3926857400090361957: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:45,584 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:[]) (logid:) Seq 2-3926857400090361957: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "9.521007537841797", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "19.291162490844727", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "29.867172241210938", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "39.73388671875", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162041712", "lastRunDuration": "45.08686065673828", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162041781", "lastRunDuration": "8064.872980117798", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162049888", "lastRunDuration": "57.860612869262695", "message": "Memory Usage within limits with current at 50.8851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162049946", "lastRunDuration": "3234.1818809509277", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162053233", "lastRunDuration": "50.071001052856445", "message": "Sufficient free space is 2571.3984375 MB"}, "lastRun": {"start": "2025-06-29 01:54:01.570477", "end": "2025-06-29 01:54:13.283695", "duration": "11.713218688964844"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161801140", "lastRunDuration": "50.86946487426758", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161801191", "lastRunDuration": "53.33065986633301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161801368", "lastRunDuration": "45.84312438964844", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161801517", "lastRunDuration": "41.30268096923828", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:50:01.137076", "end": "2025-06-29 01:50:01.603171", "duration": "0.466094970703125"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "9.521007537841797", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "19.291162490844727", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "29.867172241210938", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "39.73388671875", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162041712", "lastRunDuration": "45.08686065673828", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162041781", "lastRunDuration": "8064.872980117798", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162049888", "lastRunDuration": "57.860612869262695", "message": "Memory Usage within limits with current at 50.8851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162049946", "lastRunDuration": "3234.1818809509277", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162053233", "lastRunDuration": "50.071001052856445", "message": "Sufficient free space is 2571.3984375 MB"}, "lastRun": {"start": "2025-06-29 01:54:01.570477", "end": "2025-06-29 01:54:13.283695", "duration": "11.713218688964844"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161801140", "lastRunDuration": "50.86946487426758", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161801191", "lastRunDuration": "53.33065986633301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161801368", "lastRunDuration": "45.84312438964844", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161801517", "lastRunDuration": "41.30268096923828", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:50:01.137076", "end": "2025-06-29 01:50:01.603171", "duration": "0.466094970703125"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:45,584 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Seq 2-3926857400090361957: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 03:56:45,584 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "9.521007537841797", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "19.291162490844727", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "29.867172241210938", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "39.73388671875", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162041712", "lastRunDuration": "45.08686065673828", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162041781", "lastRunDuration": "8064.872980117798", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162049888", "lastRunDuration": "57.860612869262695", "message": "Memory Usage within limits with current at 50.8851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162049946", "lastRunDuration": "3234.1818809509277", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162053233", "lastRunDuration": "50.071001052856445", "message": "Sufficient free space is 2571.3984375 MB"}, "lastRun": {"start": "2025-06-29 01:54:01.570477", "end": "2025-06-29 01:54:13.283695", "duration": "11.713218688964844"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161801140", "lastRunDuration": "50.86946487426758", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161801191", "lastRunDuration": "53.33065986633301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161801368", "lastRunDuration": "45.84312438964844", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161801517", "lastRunDuration": "41.30268096923828", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:50:01.137076", "end": "2025-06-29 01:50:01.603171", "duration": "0.466094970703125"}}} 2025-06-29 03:56:45,618 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "9.521007537841797", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "19.291162490844727", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "29.867172241210938", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162041572", "lastRunDuration": "39.73388671875", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162041712", "lastRunDuration": "45.08686065673828", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162041781", "lastRunDuration": "8064.872980117798", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162049888", "lastRunDuration": "57.860612869262695", "message": "Memory Usage within limits with current at 50.8851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162049946", "lastRunDuration": "3234.1818809509277", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162053233", "lastRunDuration": "50.071001052856445", "message": "Sufficient free space is 2571.3984375 MB"}, "lastRun": {"start": "2025-06-29 01:54:01.570477", "end": "2025-06-29 01:54:13.283695", "duration": "11.713218688964844"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751161801140", "lastRunDuration": "50.86946487426758", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751161801191", "lastRunDuration": "53.33065986633301", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751161801368", "lastRunDuration": "45.84312438964844", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751161801517", "lastRunDuration": "41.30268096923828", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 01:50:01.137076", "end": "2025-06-29 01:50:01.603171", "duration": "0.466094970703125"}}} 2025-06-29 03:56:45,621 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found check types executed on VR basic, start: 2025-06-29 01:54:01.570477, end: 2025-06-29 01:54:13.283695, duration: 11.713218688964844 2025-06-29 03:56:45,630 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@be7960c- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 03:54:01 CEST 2025, details: service is running which took running duration (ms) 9.521007537841797 2025-06-29 03:56:45,641 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@49fbbc37- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 03:54:01 CEST 2025, details: service is running which took running duration (ms) 19.291162490844727 2025-06-29 03:56:45,653 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@345f50f8- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 03:54:01 CEST 2025, details: service is running which took running duration (ms) 29.867172241210938 2025-06-29 03:56:45,664 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1d6c08c6- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 03:54:01 CEST 2025, details: service is running which took running duration (ms) 39.73388671875 2025-06-29 03:56:45,676 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2a8cba91- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 03:54:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 45.08686065673828 2025-06-29 03:56:45,687 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@3e82a1f3- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 03:54:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8064.872980117798 2025-06-29 03:56:45,698 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7014ee4b- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 03:54:09 CEST 2025, details: Memory Usage within limits with current at 50.8851% which took running duration (ms) 57.860612869262695 2025-06-29 03:56:45,709 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@dc3f2b7- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 03:54:09 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3234.1818809509277 2025-06-29 03:56:45,719 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@42d489ac- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 03:54:13 CEST 2025, details: Sufficient free space is 2571.3984375 MB which took running duration (ms) 50.071001052856445 2025-06-29 03:56:45,720 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found check types executed on VR advanced, start: 2025-06-29 01:50:01.137076, end: 2025-06-29 01:50:01.603171, duration: 0.466094970703125 2025-06-29 03:56:45,731 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6972253c- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 03:50:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 50.86946487426758 2025-06-29 03:56:45,741 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@3ec3e8ef- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 03:50:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 53.33065986633301 2025-06-29 03:56:45,751 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2c07f3ca- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 03:50:01 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 45.84312438964844 2025-06-29 03:56:45,762 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-71595320]) (logid:c0e1f36b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4af8960f- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 03:50:01 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 41.30268096923828 2025-06-29 03:56:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:56:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:56:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7d71f67]) (logid:78f330cd) No inactive management server node found 2025-06-29 03:56:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7d71f67]) (logid:78f330cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:56:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:56:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:56:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:56:47,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:56:47,091 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:56:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:56:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:56:47,866 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac86823a]) (logid:00530c4a) No inactive management server node found 2025-06-29 03:56:47,866 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac86823a]) (logid:00530c4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:56:48,041 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-5:[ctx-f807f6dd]) (logid:7c20b1df) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 03:56:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-41106f3e]) (logid:d663488d) ClusterDRS.poll is being called at 2025-06-29T01:57:00+0000 2025-06-29 03:56:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-41106f3e]) (logid:d663488d) Removed 0 old drs migration plans 2025-06-29 03:56:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-26e547df]) (logid:69d8a23c) VM scheduler.poll is being called at 2025-06-29T01:57:00+0000 2025-06-29 03:56:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-26e547df]) (logid:69d8a23c) Got 0 scheduled jobs to be executed at 2025-06-29T01:57:00+0000 2025-06-29 03:56:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-26e547df]) (logid:69d8a23c) Cleaned up 0 VM scheduled job entries 2025-06-29 03:56:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-2d5c6bc3]) (logid:8349b362) VM Operation Thread Running 2025-06-29 03:56:48,468 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a1a58f8b]) (logid:e2377cc8) HA health check task is running... 2025-06-29 03:56:49,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9530b8be]) (logid:c9795b23) No inactive management server node found 2025-06-29 03:56:49,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9530b8be]) (logid:c9795b23) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:50,260 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8392: Seq 7-8392: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:56:50,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8392 2025-06-29 03:56:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec2bc92b]) (logid:7297deba) No inactive management server node found 2025-06-29 03:56:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec2bc92b]) (logid:7297deba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:52,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87e02ca8]) (logid:1a2e7c2f) No inactive management server node found 2025-06-29 03:56:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87e02ca8]) (logid:1a2e7c2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:52,470 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-9900701e]) (logid:63548924) HA health check task is running... 2025-06-29 03:56:53,150 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-46b4c5d2]) (logid:4ceb7368) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:56:53,159 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-46b4c5d2]) (logid:4ceb7368) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 03:56:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cffd6bd]) (logid:2d96ae65) No inactive management server node found 2025-06-29 03:56:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cffd6bd]) (logid:2d96ae65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:55,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a449e37]) (logid:8078beca) No inactive management server node found 2025-06-29 03:56:55,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a449e37]) (logid:8078beca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:56,471 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-542f2a18]) (logid:e7637005) HA health check task is running... 2025-06-29 03:56:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8d05265]) (logid:cab7aa12) No inactive management server node found 2025-06-29 03:56:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8d05265]) (logid:cab7aa12) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:56:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5973f6ba]) (logid:1ea61d71) Enabled non-edge zones available for scan: 1 2025-06-29 03:56:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5973f6ba]) (logid:1ea61d71) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:56:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5973f6ba]) (logid:1ea61d71) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:56:58,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:56:58,098 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-48112374]) (logid:fc81a870) AutoScaling Monitor is running... 2025-06-29 03:56:58,100 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Seq 2-4725683383995203675: Routed from 90520738888109 2025-06-29 03:56:58,103 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-b9e2a002]) (logid:12869c1b) Starting to remove all volume_stats rows older than [Sat Jun 28 15:56:58 CEST 2025]. 2025-06-29 03:56:58,104 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-b9e2a002]) (logid:12869c1b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:56:58 CEST 2025]. 2025-06-29 03:56:58,145 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Seq 1-4725683383995203675: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:56:58,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 03:56:58,172 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 03:56:58,174 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Seq 2-3926857400090361958: Routed from 90520738888109 2025-06-29 03:56:58,219 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Seq 2-3926857400090361958: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 03:56:58,219 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-3b9c73b5]) (logid:6b403d55) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 03:56:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-75f950cb]) (logid:3476864b) Enabled non-edge zones available for scan: 1 2025-06-29 03:56:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-75f950cb]) (logid:3476864b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:56:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-537afa27]) (logid:a100552e) No inactive management server node found 2025-06-29 03:56:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-537afa27]) (logid:a100552e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:56:58,662 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-e4d377bf]) (logid:a44c87c6) Starting to remove all vm_stats rows older than [Sat Jun 28 15:56:58 CEST 2025]. 2025-06-29 03:56:58,692 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-e4d377bf]) (logid:a44c87c6) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:56:58 CEST 2025]. 2025-06-29 03:56:59,750 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:56:59,751 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:56:59,752 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) Seq 2-4725683383995203676: Routed from 90520738888109 2025-06-29 03:56:59,795 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) Seq 1-4725683383995203676: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:56:59,797 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:56:59,798 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) Seq 2-3926857400090361959: Routed from 90520738888109 2025-06-29 03:56:59,842 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4e9fe387]) (logid:698cd3c0) Seq 2-3926857400090361959: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:56:59,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60805d26]) (logid:b2e79b50) No inactive management server node found 2025-06-29 03:56:59,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60805d26]) (logid:b2e79b50) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:00,260 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8393: Seq 7-8393: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:57:00,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8393 2025-06-29 03:57:00,472 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d0d9e748]) (logid:9f9be5f5) HA health check task is running... 2025-06-29 03:57:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34ed8e78]) (logid:a6f34d1a) No inactive management server node found 2025-06-29 03:57:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34ed8e78]) (logid:a6f34d1a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a03f4227]) (logid:852b31f6) No inactive management server node found 2025-06-29 03:57:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a03f4227]) (logid:852b31f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:03,532 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:57:03,537 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:57:03,538 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) Seq 2-4725683383995203677: Routed from 90520738888109 2025-06-29 03:57:03,591 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) Seq 1-4725683383995203677: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:57:03,630 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:57:03,631 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) Seq 2-3926857400090361960: Routed from 90520738888109 2025-06-29 03:57:03,720 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-89d4e9ab]) (logid:7df92bc0) Seq 2-3926857400090361960: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:57:04,203 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) StorageCollector is running... 2025-06-29 03:57:04,209 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:57:04,209 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:57:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:57:04,210 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3938116399158788124: Routed from 90520738888109 2025-06-29 03:57:04,288 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 4-3938116399158788124: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:57:04,292 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:57:04,292 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:57:04,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:57:04,293 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3926857400090361961: Routed from 90520738888109 2025-06-29 03:57:04,306 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3926857400090361961: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:57:04,330 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:57:04,330 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:57:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:57:04,332 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3926857400090361962: Routed from 90520738888109 2025-06-29 03:57:04,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-250484f2]) (logid:ff358eb2) No inactive management server node found 2025-06-29 03:57:04,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-250484f2]) (logid:ff358eb2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:04,426 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3926857400090361962: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:57:04,437 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:57:04,437 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:57:04,437 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:57:04,438 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-4725683383995203678: Routed from 90520738888109 2025-06-29 03:57:04,473 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6921a4eb]) (logid:735d0589) HA health check task is running... 2025-06-29 03:57:04,482 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 1-4725683383995203678: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:57:04,485 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:57:04,485 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:57:04,486 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:57:04,487 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3926857400090361963: Routed from 90520738888109 2025-06-29 03:57:04,532 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-49b8d3d9]) (logid:30c39e98) Seq 2-3926857400090361963: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:57:05,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-888e2361]) (logid:f8a19365) No inactive management server node found 2025-06-29 03:57:05,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-888e2361]) (logid:f8a19365) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74a6e99]) (logid:2a71372c) No inactive management server node found 2025-06-29 03:57:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74a6e99]) (logid:2a71372c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:08,475 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-8e0376ec]) (logid:b359998a) HA health check task is running... 2025-06-29 03:57:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8718edf]) (logid:78bda80e) No inactive management server node found 2025-06-29 03:57:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8718edf]) (logid:78bda80e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:10,261 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8394: Seq 7-8394: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:57:10,283 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8394 2025-06-29 03:57:10,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a39a1aa]) (logid:1591d36a) No inactive management server node found 2025-06-29 03:57:10,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a39a1aa]) (logid:1591d36a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ee7d29e]) (logid:635a75e4) No inactive management server node found 2025-06-29 03:57:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ee7d29e]) (logid:635a75e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:12,476 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a6bc4056]) (logid:0604c319) HA health check task is running... 2025-06-29 03:57:13,002 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-a312c991]) (logid:ba34261f) Checking if any host reservation can be released ... 2025-06-29 03:57:13,008 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-a312c991]) (logid:ba34261f) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-a312c991]) (logid:ba34261f) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-a312c991]) (logid:ba34261f) Done running HostReservationReleaseChecker ... 2025-06-29 03:57:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a05e6bc9]) (logid:be26bbf0) Found 0 routers to update status. 2025-06-29 03:57:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a05e6bc9]) (logid:be26bbf0) Found 0 VPC's to update Redundant State. 2025-06-29 03:57:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a05e6bc9]) (logid:be26bbf0) Found 0 networks to update RvR status. 2025-06-29 03:57:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Running Capacity Checker ... 2025-06-29 03:57:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) recalculating system capacity 2025-06-29 03:57:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Executing cpu/ram capacity update 2025-06-29 03:57:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:13,181 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:13,187 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 03:57:13,187 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 03:57:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 03:57:13,201 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:13,211 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 03:57:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 03:57:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-20-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 03:57:13,223 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Done executing cpu/ram capacity update 2025-06-29 03:57:13,223 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Executing storage capacity update 2025-06-29 03:57:13,231 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:57:13,232 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 03:57:13,232 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 03:57:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 03:57:13,242 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 03:57:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 03:57:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 03:57:13,246 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 03:57:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:57:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 03:57:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 03:57:13,258 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 03:57:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 03:57:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 03:57:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 03:57:13,271 DEBUG [c.c.s.StorageManagerImpl] (pool-21-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 03:57:13,272 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Done executing storage capacity update 2025-06-29 03:57:13,272 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Executing capacity updates for public ip and Vlans 2025-06-29 03:57:13,285 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Done capacity updates for public ip and Vlans 2025-06-29 03:57:13,285 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Executing capacity updates for private ip 2025-06-29 03:57:13,289 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Done executing capacity updates for private ip 2025-06-29 03:57:13,289 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Done recalculating system capacity 2025-06-29 03:57:13,326 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bf8fdda9]) (logid:bf0e00ae) Done running Capacity Checker ... 2025-06-29 03:57:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e6a9c4f]) (logid:0e1f0d21) No inactive management server node found 2025-06-29 03:57:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e6a9c4f]) (logid:0e1f0d21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d652ddf]) (logid:aeb74627) No inactive management server node found 2025-06-29 03:57:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d652ddf]) (logid:aeb74627) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99128817]) (logid:446a00a5) No inactive management server node found 2025-06-29 03:57:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99128817]) (logid:446a00a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:16,477 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cb9c1cb2]) (logid:4382b2e4) HA health check task is running... 2025-06-29 03:57:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9a657b2]) (logid:1cdae2f7) No inactive management server node found 2025-06-29 03:57:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9a657b2]) (logid:1cdae2f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:19,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29bb2eb3]) (logid:b0ba81e7) No inactive management server node found 2025-06-29 03:57:19,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29bb2eb3]) (logid:b0ba81e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:20,262 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8395: Seq 7-8395: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:57:20,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8395 2025-06-29 03:57:20,479 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-84425ad9]) (logid:10389433) HA health check task is running... 2025-06-29 03:57:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f27ae2ed]) (logid:845bd3a6) No inactive management server node found 2025-06-29 03:57:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f27ae2ed]) (logid:845bd3a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:22,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b87b21d]) (logid:7310aa67) No inactive management server node found 2025-06-29 03:57:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b87b21d]) (logid:7310aa67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:22,715 DEBUG [c.c.a.ApiServlet] (qtp1513608173-23:[ctx-30d9fd98]) (logid:584f605a) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&response=json&signature=Gww38REgqUfBtJAfzvSTiRfV%2Fnc%3D 2025-06-29 03:57:22,728 DEBUG [c.c.a.ApiServer] (qtp1513608173-23:[ctx-30d9fd98, ctx-ba3abf35]) (logid:584f605a) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:22,733 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-23:[ctx-30d9fd98, ctx-ba3abf35]) (logid:584f605a) Account for user id 8d323c24-0fc7-4072-9caa-66dd3c7d6562 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:22,733 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-23:[ctx-30d9fd98, ctx-ba3abf35]) (logid:584f605a) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:22,734 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-23:[ctx-30d9fd98, ctx-ba3abf35]) (logid:584f605a) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:22,788 DEBUG [c.c.a.ApiServlet] (qtp1513608173-23:[ctx-30d9fd98, ctx-ba3abf35, ctx-e7053619]) (logid:584f605a) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&response=json&signature=Gww38REgqUfBtJAfzvSTiRfV%2Fnc%3D 2025-06-29 03:57:22,801 DEBUG [c.c.a.ApiServlet] (qtp1513608173-18:[ctx-80f60597]) (logid:24dc0c75) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=y5g4SVP4LaYlcIpY%2FvDpvphKmmI%3D 2025-06-29 03:57:22,809 DEBUG [c.c.a.ApiServer] (qtp1513608173-18:[ctx-80f60597, ctx-260ac9d9]) (logid:24dc0c75) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:22,813 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-18:[ctx-80f60597, ctx-260ac9d9]) (logid:24dc0c75) Account for user id 8baba5bc-275a-4a1b-a609-219210453d91 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:22,813 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-18:[ctx-80f60597, ctx-260ac9d9]) (logid:24dc0c75) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:22,813 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-18:[ctx-80f60597, ctx-260ac9d9]) (logid:24dc0c75) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:22,833 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-18:[ctx-80f60597, ctx-260ac9d9, ctx-0108575c]) (logid:24dc0c75) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"6eb60fe9-5f1f-4db9-aaf0-e020f7f34db0"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:22,874 DEBUG [c.c.a.ApiServlet] (qtp1513608173-18:[ctx-80f60597, ctx-260ac9d9, ctx-0108575c]) (logid:24dc0c75) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=y5g4SVP4LaYlcIpY%2FvDpvphKmmI%3D 2025-06-29 03:57:22,878 DEBUG [c.c.a.ApiServlet] (qtp1513608173-17:[ctx-e879e728]) (logid:d2cce8cc) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&name=hgx&response=json&signature=yxTg%2FjSKCXlAX%2FQq8iOGWsLk4Nc%3D 2025-06-29 03:57:22,886 DEBUG [c.c.a.ApiServer] (qtp1513608173-17:[ctx-e879e728, ctx-2a07cd12]) (logid:d2cce8cc) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:22,889 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-17:[ctx-e879e728, ctx-2a07cd12]) (logid:d2cce8cc) Account for user id c849b073-9eeb-4932-be64-1bf71dd4c221 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:22,889 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-17:[ctx-e879e728, ctx-2a07cd12]) (logid:d2cce8cc) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:22,889 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-17:[ctx-e879e728, ctx-2a07cd12]) (logid:d2cce8cc) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:22,901 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-17:[ctx-e879e728, ctx-2a07cd12, ctx-88ee0984]) (logid:d2cce8cc) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"53448aa7-b43c-4334-b28f-666d7558b142"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:22,969 DEBUG [c.c.a.ApiServlet] (qtp1513608173-17:[ctx-e879e728, ctx-2a07cd12, ctx-88ee0984]) (logid:d2cce8cc) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&name=hgx&response=json&signature=yxTg%2FjSKCXlAX%2FQq8iOGWsLk4Nc%3D 2025-06-29 03:57:22,975 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-92e36948]) (logid:721734de) ===START=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=KDuMtkJSp2RPd2F44ApT5M%2F4MtY%3D 2025-06-29 03:57:22,981 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-92e36948, ctx-c574a1ec]) (logid:721734de) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:22,984 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-92e36948, ctx-c574a1ec]) (logid:721734de) Account for user id ce89f959-c2d6-4349-9433-bb9f5f84de21 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:22,984 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-92e36948, ctx-c574a1ec]) (logid:721734de) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:22,984 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-92e36948, ctx-c574a1ec]) (logid:721734de) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:22,990 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-21:[ctx-92e36948, ctx-c574a1ec, ctx-d06cfb24]) (logid:721734de) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"3aa5d272-a701-4490-957d-0e78acc344d0"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,017 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-92e36948, ctx-c574a1ec, ctx-d06cfb24]) (logid:721734de) ===END=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=KDuMtkJSp2RPd2F44ApT5M%2F4MtY%3D 2025-06-29 03:57:23,022 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-53772994]) (logid:22b1614a) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&level=0&listall=true&name=ROOT&response=json&signature=qLk1LalACDcu169h7XWVQORa6V4%3D 2025-06-29 03:57:23,032 DEBUG [c.c.a.ApiServer] (qtp1513608173-25:[ctx-53772994, ctx-0ebb3444]) (logid:22b1614a) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,036 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-53772994, ctx-0ebb3444]) (logid:22b1614a) Account for user id 5160d83a-80ff-4be2-bf5e-261a2158a0ff is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,036 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-53772994, ctx-0ebb3444]) (logid:22b1614a) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,036 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-25:[ctx-53772994, ctx-0ebb3444]) (logid:22b1614a) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,044 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-25:[ctx-53772994, ctx-0ebb3444, ctx-9dee8c3d]) (logid:22b1614a) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"d1fe2453-862c-4617-9954-ed66d9df8353"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,066 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-53772994, ctx-0ebb3444, ctx-9dee8c3d]) (logid:22b1614a) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&level=0&listall=true&name=ROOT&response=json&signature=qLk1LalACDcu169h7XWVQORa6V4%3D 2025-06-29 03:57:23,071 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-89b54859]) (logid:016a0713) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=6b11f146-3810-4f53-a521-358f96b1a404&name=hgx&response=json&signature=yUZXO4RmWzfXTI55DBv2Nq57tZQ%3D 2025-06-29 03:57:23,080 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-89b54859, ctx-0f283426]) (logid:016a0713) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,083 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-89b54859, ctx-0f283426]) (logid:016a0713) Account for user id b481a6ff-b5c9-4d54-ba11-fd5cef3547b5 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,083 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-89b54859, ctx-0f283426]) (logid:016a0713) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,083 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-89b54859, ctx-0f283426]) (logid:016a0713) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,094 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-20:[ctx-89b54859, ctx-0f283426, ctx-3f9ae65c]) (logid:016a0713) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"7a5405ad-4000-4c33-bb2f-61c3e0476a27"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,134 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-89b54859, ctx-0f283426, ctx-3f9ae65c]) (logid:016a0713) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=6b11f146-3810-4f53-a521-358f96b1a404&name=hgx&response=json&signature=yUZXO4RmWzfXTI55DBv2Nq57tZQ%3D 2025-06-29 03:57:23,140 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-f42d3065]) (logid:53348c6a) ===START=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=57058d2c-ae3c-482c-9bab-507a7fc046af&listall=true&response=json&signature=ndys0lrhbFdSQe4UZ9LQNgstIAU%3D 2025-06-29 03:57:23,147 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-f42d3065, ctx-847967e2]) (logid:53348c6a) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,150 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-f42d3065, ctx-847967e2]) (logid:53348c6a) Account for user id 5dbe84d1-36fe-4c90-8b76-38b4996c04ab is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,150 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-f42d3065, ctx-847967e2]) (logid:53348c6a) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,150 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-f42d3065, ctx-847967e2]) (logid:53348c6a) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,163 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-21:[ctx-f42d3065, ctx-847967e2, ctx-a7871e43]) (logid:53348c6a) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"6c5b3564-f565-4669-be88-63676d16098e"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,179 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-f42d3065, ctx-847967e2, ctx-a7871e43]) (logid:53348c6a) ===END=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=57058d2c-ae3c-482c-9bab-507a7fc046af&listall=true&response=json&signature=ndys0lrhbFdSQe4UZ9LQNgstIAU%3D 2025-06-29 03:57:23,186 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-9e5e0774]) (logid:f3b7ad7c) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=getUserKeys&id=57058d2c-ae3c-482c-9bab-507a7fc046af&response=json&signature=zsPoEGb3xx0uuHT0GJstct2Sf64%3D 2025-06-29 03:57:23,195 DEBUG [c.c.a.ApiServer] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b]) (logid:f3b7ad7c) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,200 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b]) (logid:f3b7ad7c) Account for user id b955ca0f-5fe6-463f-92d7-659d0bbf63a8 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,200 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b]) (logid:f3b7ad7c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,200 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b]) (logid:f3b7ad7c) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,236 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b, ctx-dc935645]) (logid:f3b7ad7c) Access granted to User {"id":5,"username":"jdhirst","uuid":"b955ca0f-5fe6-463f-92d7-659d0bbf63a8"}.to Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]by DomainChecker 2025-06-29 03:57:23,237 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b, ctx-dc935645]) (logid:f3b7ad7c) Verifying whether the caller has the correct privileges based on the user's role type and API permissions: User {"id":5,"username":"jdhirst","uuid":"57058d2c-ae3c-482c-9bab-507a7fc046af"}. 2025-06-29 03:57:23,247 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-9e5e0774, ctx-7f70b54b, ctx-dc935645]) (logid:f3b7ad7c) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=getUserKeys&id=57058d2c-ae3c-482c-9bab-507a7fc046af&response=json&signature=zsPoEGb3xx0uuHT0GJstct2Sf64%3D 2025-06-29 03:57:23,251 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-0f79b173]) (logid:1d416d35) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&level=0&listall=true&name=ROOT&response=json&signature=SV6HxQ5iVyX1EHsG6tFjjzT8LZE%3D 2025-06-29 03:57:23,258 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-0f79b173, ctx-0a1086f9]) (logid:1d416d35) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,261 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-0f79b173, ctx-0a1086f9]) (logid:1d416d35) Account for user id 9d504a18-8633-4e43-b7aa-7a8d4f93c01d is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,261 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-0f79b173, ctx-0a1086f9]) (logid:1d416d35) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,261 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-0f79b173, ctx-0a1086f9]) (logid:1d416d35) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,287 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-0f79b173, ctx-0a1086f9, ctx-229af3ae]) (logid:1d416d35) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&level=0&listall=true&name=ROOT&response=json&signature=SV6HxQ5iVyX1EHsG6tFjjzT8LZE%3D 2025-06-29 03:57:23,291 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-f1390e7c]) (logid:038ca242) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&name=hgx&response=json&signature=yxTg%2FjSKCXlAX%2FQq8iOGWsLk4Nc%3D 2025-06-29 03:57:23,301 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-f1390e7c, ctx-ec927a5b]) (logid:038ca242) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,305 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-f1390e7c, ctx-ec927a5b]) (logid:038ca242) Account for user id 338d6478-8049-45a0-ae80-8b5f527db451 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,305 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-f1390e7c, ctx-ec927a5b]) (logid:038ca242) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,305 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-f1390e7c, ctx-ec927a5b]) (logid:038ca242) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,336 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-21:[ctx-f1390e7c, ctx-ec927a5b, ctx-8c81874f]) (logid:038ca242) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"31ed9476-fcff-4e0d-abb8-5c4559ea057c"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,394 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-f1390e7c, ctx-ec927a5b, ctx-8c81874f]) (logid:038ca242) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&name=hgx&response=json&signature=yxTg%2FjSKCXlAX%2FQq8iOGWsLk4Nc%3D 2025-06-29 03:57:23,399 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-231ebcf0]) (logid:7e9d7345) ===START=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=KDuMtkJSp2RPd2F44ApT5M%2F4MtY%3D 2025-06-29 03:57:23,410 DEBUG [c.c.a.ApiServer] (qtp1513608173-25:[ctx-231ebcf0, ctx-f93ee4cb]) (logid:7e9d7345) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,413 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-231ebcf0, ctx-f93ee4cb]) (logid:7e9d7345) Account for user id 6b6d3f63-9501-4164-b05d-ed2e840b5287 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,413 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-231ebcf0, ctx-f93ee4cb]) (logid:7e9d7345) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,413 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-25:[ctx-231ebcf0, ctx-f93ee4cb]) (logid:7e9d7345) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,419 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-25:[ctx-231ebcf0, ctx-f93ee4cb, ctx-dcbf534f]) (logid:7e9d7345) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"c67c09cd-6fe7-44a8-a7cf-57de48fac4c4"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,434 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-231ebcf0, ctx-f93ee4cb, ctx-dcbf534f]) (logid:7e9d7345) ===END=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=KDuMtkJSp2RPd2F44ApT5M%2F4MtY%3D 2025-06-29 03:57:23,439 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-3764f44a]) (logid:c7725fac) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&level=0&listall=true&name=ROOT&response=json&signature=qLk1LalACDcu169h7XWVQORa6V4%3D 2025-06-29 03:57:23,445 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-3764f44a, ctx-a7dd2a0a]) (logid:c7725fac) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,447 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-3764f44a, ctx-a7dd2a0a]) (logid:c7725fac) Account for user id 92514468-2126-4140-9bfa-0a738a507f95 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,447 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-3764f44a, ctx-a7dd2a0a]) (logid:c7725fac) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,447 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-3764f44a, ctx-a7dd2a0a]) (logid:c7725fac) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,453 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-20:[ctx-3764f44a, ctx-a7dd2a0a, ctx-d09d725a]) (logid:c7725fac) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"31b82bd3-2bde-454e-bdbc-dd1f62955832"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,469 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-3764f44a, ctx-a7dd2a0a, ctx-d09d725a]) (logid:c7725fac) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&level=0&listall=true&name=ROOT&response=json&signature=qLk1LalACDcu169h7XWVQORa6V4%3D 2025-06-29 03:57:23,473 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-4743467c]) (logid:4d774b6e) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=6b11f146-3810-4f53-a521-358f96b1a404&name=hgx&response=json&signature=yUZXO4RmWzfXTI55DBv2Nq57tZQ%3D 2025-06-29 03:57:23,482 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-4743467c, ctx-3dc1c169]) (logid:4d774b6e) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,486 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-4743467c, ctx-3dc1c169]) (logid:4d774b6e) Account for user id ac3038ae-6d85-44fe-b537-c071b27cbb3a is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,486 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-4743467c, ctx-3dc1c169]) (logid:4d774b6e) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,486 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-4743467c, ctx-3dc1c169]) (logid:4d774b6e) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,500 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-21:[ctx-4743467c, ctx-3dc1c169, ctx-4a132d43]) (logid:4d774b6e) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"ea74e5bd-d0cb-4815-9682-2932ff09343a"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,546 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-4743467c, ctx-3dc1c169, ctx-4a132d43]) (logid:4d774b6e) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=6b11f146-3810-4f53-a521-358f96b1a404&name=hgx&response=json&signature=yUZXO4RmWzfXTI55DBv2Nq57tZQ%3D 2025-06-29 03:57:23,550 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-5f6ff2cf]) (logid:8f1b8b20) ===START=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=57058d2c-ae3c-482c-9bab-507a7fc046af&listall=true&response=json&signature=ndys0lrhbFdSQe4UZ9LQNgstIAU%3D 2025-06-29 03:57:23,557 DEBUG [c.c.a.ApiServer] (qtp1513608173-25:[ctx-5f6ff2cf, ctx-88ebc237]) (logid:8f1b8b20) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,560 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-5f6ff2cf, ctx-88ebc237]) (logid:8f1b8b20) Account for user id 798d6f43-90e5-42fb-aee0-ca1273fb69d6 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,560 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-5f6ff2cf, ctx-88ebc237]) (logid:8f1b8b20) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,560 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-25:[ctx-5f6ff2cf, ctx-88ebc237]) (logid:8f1b8b20) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,571 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-25:[ctx-5f6ff2cf, ctx-88ebc237, ctx-4b5741b2]) (logid:8f1b8b20) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"ac080dba-92fe-4f76-ae8b-459e0ad72e61"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,582 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-5f6ff2cf, ctx-88ebc237, ctx-4b5741b2]) (logid:8f1b8b20) ===END=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=57058d2c-ae3c-482c-9bab-507a7fc046af&listall=true&response=json&signature=ndys0lrhbFdSQe4UZ9LQNgstIAU%3D 2025-06-29 03:57:23,586 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-3ab6409e]) (logid:205f6d3d) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=getUserKeys&id=57058d2c-ae3c-482c-9bab-507a7fc046af&response=json&signature=zsPoEGb3xx0uuHT0GJstct2Sf64%3D 2025-06-29 03:57:23,592 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf]) (logid:205f6d3d) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,594 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf]) (logid:205f6d3d) Account for user id 72f2d95e-20af-4931-93d7-d3a7fd06ff67 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,594 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf]) (logid:205f6d3d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,594 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf]) (logid:205f6d3d) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,605 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf, ctx-571c6399]) (logid:205f6d3d) Access granted to User {"id":5,"username":"jdhirst","uuid":"72f2d95e-20af-4931-93d7-d3a7fd06ff67"}.to Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]by DomainChecker 2025-06-29 03:57:23,606 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf, ctx-571c6399]) (logid:205f6d3d) Verifying whether the caller has the correct privileges based on the user's role type and API permissions: User {"id":5,"username":"jdhirst","uuid":"57058d2c-ae3c-482c-9bab-507a7fc046af"}. 2025-06-29 03:57:23,611 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-3ab6409e, ctx-c80a70cf, ctx-571c6399]) (logid:205f6d3d) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=getUserKeys&id=57058d2c-ae3c-482c-9bab-507a7fc046af&response=json&signature=zsPoEGb3xx0uuHT0GJstct2Sf64%3D 2025-06-29 03:57:23,619 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-7fee43b7]) (logid:57bc42d2) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&name=az1&response=json&signature=XN5EEYWqBStqy1BUQ8ouCLnLm5A%3D 2025-06-29 03:57:23,626 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-7fee43b7, ctx-b546c9d1]) (logid:57bc42d2) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,629 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-7fee43b7, ctx-b546c9d1]) (logid:57bc42d2) Account for user id a7b45a39-3c3b-42bc-a7c4-bf507e4374f6 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,629 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-7fee43b7, ctx-b546c9d1]) (logid:57bc42d2) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,629 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-7fee43b7, ctx-b546c9d1]) (logid:57bc42d2) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,678 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-7fee43b7, ctx-b546c9d1, ctx-5a88d9ae]) (logid:57bc42d2) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&name=az1&response=json&signature=XN5EEYWqBStqy1BUQ8ouCLnLm5A%3D 2025-06-29 03:57:23,683 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-9ea5a578]) (logid:13186883) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&id=55438e7b-e5f2-4988-bbef-af0a21044016&response=json&signature=YsnNklIUrghmqPBnP7vFqEj%2FbbM%3D 2025-06-29 03:57:23,688 DEBUG [c.c.a.ApiServer] (qtp1513608173-25:[ctx-9ea5a578, ctx-3dcdd811]) (logid:13186883) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,691 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-9ea5a578, ctx-3dcdd811]) (logid:13186883) Account for user id 29796871-0d92-4690-95e8-726b32ea212f is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,691 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-25:[ctx-9ea5a578, ctx-3dcdd811]) (logid:13186883) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,691 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-25:[ctx-9ea5a578, ctx-3dcdd811]) (logid:13186883) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,714 DEBUG [c.c.a.ApiServlet] (qtp1513608173-25:[ctx-9ea5a578, ctx-3dcdd811, ctx-aebf074f]) (logid:13186883) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&id=55438e7b-e5f2-4988-bbef-af0a21044016&response=json&signature=YsnNklIUrghmqPBnP7vFqEj%2FbbM%3D 2025-06-29 03:57:23,718 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-61b57778]) (logid:80d173c8) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&keyword=pvnet1&response=json&signature=gn%2F3PSTy7k3lY7Ss74VnGlaPnrU%3D 2025-06-29 03:57:23,725 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-61b57778, ctx-ff130c35]) (logid:80d173c8) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,727 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-61b57778, ctx-ff130c35]) (logid:80d173c8) Account for user id cb105b53-5397-48ef-bac7-140502018dce is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,727 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-61b57778, ctx-ff130c35]) (logid:80d173c8) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,727 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-61b57778, ctx-ff130c35]) (logid:80d173c8) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,814 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-61b57778, ctx-ff130c35, ctx-9471a5e8]) (logid:80d173c8) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&keyword=pvnet1&response=json&signature=gn%2F3PSTy7k3lY7Ss74VnGlaPnrU%3D 2025-06-29 03:57:23,819 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-2e6b64bd]) (logid:1c63089c) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&id=564b4e1d-cbd0-431e-964d-5e15e592ca06&response=json&signature=c1laI4ClExqCugfgAsDqmJ2v7gI%3D 2025-06-29 03:57:23,829 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-2e6b64bd, ctx-b3dfdc70]) (logid:1c63089c) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,832 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-2e6b64bd, ctx-b3dfdc70]) (logid:1c63089c) Account for user id 9a9a1fab-314c-4faf-b028-88e98628115f is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,833 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-2e6b64bd, ctx-b3dfdc70]) (logid:1c63089c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,833 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-2e6b64bd, ctx-b3dfdc70]) (logid:1c63089c) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ebe739e]) (logid:5448df49) No inactive management server node found 2025-06-29 03:57:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ebe739e]) (logid:5448df49) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:23,900 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-2e6b64bd, ctx-b3dfdc70, ctx-575ddd80]) (logid:1c63089c) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&id=564b4e1d-cbd0-431e-964d-5e15e592ca06&response=json&signature=c1laI4ClExqCugfgAsDqmJ2v7gI%3D 2025-06-29 03:57:23,907 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-cb92eb54]) (logid:e8b5f6d7) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&level=0&listall=true&name=ROOT&response=json&signature=SV6HxQ5iVyX1EHsG6tFjjzT8LZE%3D 2025-06-29 03:57:23,913 DEBUG [c.c.a.ApiServer] (qtp1513608173-26:[ctx-cb92eb54, ctx-dea07f8f]) (logid:e8b5f6d7) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,915 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-cb92eb54, ctx-dea07f8f]) (logid:e8b5f6d7) Account for user id d5830650-d5eb-4c64-be4b-5fa57d84fc73 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,915 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-cb92eb54, ctx-dea07f8f]) (logid:e8b5f6d7) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,915 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-26:[ctx-cb92eb54, ctx-dea07f8f]) (logid:e8b5f6d7) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,934 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-cb92eb54, ctx-dea07f8f, ctx-ebb1f9bf]) (logid:e8b5f6d7) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&level=0&listall=true&name=ROOT&response=json&signature=SV6HxQ5iVyX1EHsG6tFjjzT8LZE%3D 2025-06-29 03:57:23,939 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-e6f5c64a]) (logid:c9042792) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&name=hgx&response=json&signature=yxTg%2FjSKCXlAX%2FQq8iOGWsLk4Nc%3D 2025-06-29 03:57:23,944 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-e6f5c64a, ctx-11315c6e]) (logid:c9042792) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:23,947 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-e6f5c64a, ctx-11315c6e]) (logid:c9042792) Account for user id 3b87ccc2-6104-458f-b32c-43f4ce43f2a6 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:23,947 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-e6f5c64a, ctx-11315c6e]) (logid:c9042792) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:23,947 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-e6f5c64a, ctx-11315c6e]) (logid:c9042792) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:23,955 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-20:[ctx-e6f5c64a, ctx-11315c6e, ctx-7a715f44]) (logid:c9042792) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"052c4898-8254-4388-b983-b8e3beaf456d"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:23,999 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-e6f5c64a, ctx-11315c6e, ctx-7a715f44]) (logid:c9042792) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&name=hgx&response=json&signature=yxTg%2FjSKCXlAX%2FQq8iOGWsLk4Nc%3D 2025-06-29 03:57:24,010 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-febae0ed]) (logid:7425f3b1) ===START=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=KDuMtkJSp2RPd2F44ApT5M%2F4MtY%3D 2025-06-29 03:57:24,024 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-febae0ed, ctx-931c0863]) (logid:7425f3b1) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,027 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-febae0ed, ctx-931c0863]) (logid:7425f3b1) Account for user id ea3b9717-52b3-4c26-a8dc-ab1d73abf86e is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,027 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-febae0ed, ctx-931c0863]) (logid:7425f3b1) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,027 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-febae0ed, ctx-931c0863]) (logid:7425f3b1) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,032 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-21:[ctx-febae0ed, ctx-931c0863, ctx-2351bac2]) (logid:7425f3b1) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"9ef0cc7b-047c-4665-84b1-6d76d760e396"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:24,049 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-febae0ed, ctx-931c0863, ctx-2351bac2]) (logid:7425f3b1) ===END=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&listall=true&response=json&signature=KDuMtkJSp2RPd2F44ApT5M%2F4MtY%3D 2025-06-29 03:57:24,054 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-68ea66c9]) (logid:a5a588ed) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&level=0&listall=true&name=ROOT&response=json&signature=qLk1LalACDcu169h7XWVQORa6V4%3D 2025-06-29 03:57:24,060 DEBUG [c.c.a.ApiServer] (qtp1513608173-26:[ctx-68ea66c9, ctx-d0c3c944]) (logid:a5a588ed) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,064 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-68ea66c9, ctx-d0c3c944]) (logid:a5a588ed) Account for user id 47fbef3c-f2ae-4d7e-a6ee-38ff8656c0c2 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,064 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-68ea66c9, ctx-d0c3c944]) (logid:a5a588ed) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,064 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-26:[ctx-68ea66c9, ctx-d0c3c944]) (logid:a5a588ed) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,072 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-26:[ctx-68ea66c9, ctx-d0c3c944, ctx-14189f25]) (logid:a5a588ed) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"f5eff458-c2b5-4e56-b3ad-1330202ee4c0"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:24,090 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-68ea66c9, ctx-d0c3c944, ctx-14189f25]) (logid:a5a588ed) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listDomains&id=01426049-6d14-11ef-8312-5254007c79ad&level=0&listall=true&name=ROOT&response=json&signature=qLk1LalACDcu169h7XWVQORa6V4%3D 2025-06-29 03:57:24,094 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-8905c846]) (logid:bcb965b4) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=6b11f146-3810-4f53-a521-358f96b1a404&name=hgx&response=json&signature=yUZXO4RmWzfXTI55DBv2Nq57tZQ%3D 2025-06-29 03:57:24,100 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-8905c846, ctx-cb148b26]) (logid:bcb965b4) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,103 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-8905c846, ctx-cb148b26]) (logid:bcb965b4) Account for user id e380a4fa-1930-407a-aaa5-0d0bb8c21228 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,103 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-8905c846, ctx-cb148b26]) (logid:bcb965b4) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,103 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-8905c846, ctx-cb148b26]) (logid:bcb965b4) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,115 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-20:[ctx-8905c846, ctx-cb148b26, ctx-568a1961]) (logid:bcb965b4) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"8542a6bf-fad3-49b2-8be5-464ca12d1d38"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:24,162 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-8905c846, ctx-cb148b26, ctx-568a1961]) (logid:bcb965b4) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listAccounts&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=6b11f146-3810-4f53-a521-358f96b1a404&name=hgx&response=json&signature=yUZXO4RmWzfXTI55DBv2Nq57tZQ%3D 2025-06-29 03:57:24,168 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-ae0320f7]) (logid:5582876c) ===START=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=57058d2c-ae3c-482c-9bab-507a7fc046af&listall=true&response=json&signature=ndys0lrhbFdSQe4UZ9LQNgstIAU%3D 2025-06-29 03:57:24,176 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-ae0320f7, ctx-8bdef38a]) (logid:5582876c) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,185 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-ae0320f7, ctx-8bdef38a]) (logid:5582876c) Account for user id 1b19e03d-c62f-43c9-bc5c-900896c5199c is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,185 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-ae0320f7, ctx-8bdef38a]) (logid:5582876c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,185 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-ae0320f7, ctx-8bdef38a]) (logid:5582876c) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,197 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-21:[ctx-ae0320f7, ctx-8bdef38a, ctx-d2508ce1]) (logid:5582876c) Access granted to Account [{"accountName":"hgx","id":5,"uuid":"3882f040-c6a2-4f06-a943-ddbc9db89afc"}] to Domain {"id":1,"name":"ROOT","path":"\/","uuid":"01426049-6d14-11ef-8312-5254007c79ad"} by AffinityGroupAccessChecker 2025-06-29 03:57:24,216 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-ae0320f7, ctx-8bdef38a, ctx-d2508ce1]) (logid:5582876c) ===END=== 127.0.0.1 -- GET account=hgx&apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listUsers&domainid=01426049-6d14-11ef-8312-5254007c79ad&id=57058d2c-ae3c-482c-9bab-507a7fc046af&listall=true&response=json&signature=ndys0lrhbFdSQe4UZ9LQNgstIAU%3D 2025-06-29 03:57:24,221 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-7d4ca57c]) (logid:417caa78) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=getUserKeys&id=57058d2c-ae3c-482c-9bab-507a7fc046af&response=json&signature=zsPoEGb3xx0uuHT0GJstct2Sf64%3D 2025-06-29 03:57:24,229 DEBUG [c.c.a.ApiServer] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8]) (logid:417caa78) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,234 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8]) (logid:417caa78) Account for user id bbd993f7-a6b6-48b6-ab56-5f6b59a9f91a is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,234 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8]) (logid:417caa78) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,234 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8]) (logid:417caa78) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,251 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8, ctx-5dfa7057]) (logid:417caa78) Access granted to User {"id":5,"username":"jdhirst","uuid":"bbd993f7-a6b6-48b6-ab56-5f6b59a9f91a"}.to Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]by DomainChecker 2025-06-29 03:57:24,252 DEBUG [c.c.u.AccountManagerImpl] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8, ctx-5dfa7057]) (logid:417caa78) Verifying whether the caller has the correct privileges based on the user's role type and API permissions: User {"id":5,"username":"jdhirst","uuid":"57058d2c-ae3c-482c-9bab-507a7fc046af"}. 2025-06-29 03:57:24,258 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-7d4ca57c, ctx-ac9512d8, ctx-5dfa7057]) (logid:417caa78) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=getUserKeys&id=57058d2c-ae3c-482c-9bab-507a7fc046af&response=json&signature=zsPoEGb3xx0uuHT0GJstct2Sf64%3D 2025-06-29 03:57:24,262 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-4cec1470]) (logid:2ebbc7ee) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&name=az1&response=json&signature=XN5EEYWqBStqy1BUQ8ouCLnLm5A%3D 2025-06-29 03:57:24,270 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-4cec1470, ctx-9314a1d0]) (logid:2ebbc7ee) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,273 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-4cec1470, ctx-9314a1d0]) (logid:2ebbc7ee) Account for user id 1725587c-db18-40f6-b1c9-fd2da23c119e is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,273 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-4cec1470, ctx-9314a1d0]) (logid:2ebbc7ee) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,273 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-4cec1470, ctx-9314a1d0]) (logid:2ebbc7ee) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,298 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-4cec1470, ctx-9314a1d0, ctx-a0417f35]) (logid:2ebbc7ee) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&name=az1&response=json&signature=XN5EEYWqBStqy1BUQ8ouCLnLm5A%3D 2025-06-29 03:57:24,305 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-702bc524]) (logid:0f567586) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&id=55438e7b-e5f2-4988-bbef-af0a21044016&response=json&signature=YsnNklIUrghmqPBnP7vFqEj%2FbbM%3D 2025-06-29 03:57:24,328 DEBUG [c.c.a.ApiServer] (qtp1513608173-21:[ctx-702bc524, ctx-bc506419]) (logid:0f567586) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,341 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-702bc524, ctx-bc506419]) (logid:0f567586) Account for user id 04fd5805-1337-492e-9ec4-5e495e698e36 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,341 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-21:[ctx-702bc524, ctx-bc506419]) (logid:0f567586) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,341 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-21:[ctx-702bc524, ctx-bc506419]) (logid:0f567586) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,373 DEBUG [c.c.a.ApiServlet] (qtp1513608173-21:[ctx-702bc524, ctx-bc506419, ctx-f5e54a23]) (logid:0f567586) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listZones&id=55438e7b-e5f2-4988-bbef-af0a21044016&response=json&signature=YsnNklIUrghmqPBnP7vFqEj%2FbbM%3D 2025-06-29 03:57:24,378 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-15d53348]) (logid:4cc59fb8) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&keyword=pvnet1&response=json&signature=gn%2F3PSTy7k3lY7Ss74VnGlaPnrU%3D 2025-06-29 03:57:24,386 DEBUG [c.c.a.ApiServer] (qtp1513608173-26:[ctx-15d53348, ctx-c565ff04]) (logid:4cc59fb8) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,389 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-15d53348, ctx-c565ff04]) (logid:4cc59fb8) Account for user id 9f1cda46-fa70-4899-bdf6-a9d97b6e7505 is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,389 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-26:[ctx-15d53348, ctx-c565ff04]) (logid:4cc59fb8) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,389 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-26:[ctx-15d53348, ctx-c565ff04]) (logid:4cc59fb8) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,444 DEBUG [c.c.a.ApiServlet] (qtp1513608173-26:[ctx-15d53348, ctx-c565ff04, ctx-323512bd]) (logid:4cc59fb8) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&keyword=pvnet1&response=json&signature=gn%2F3PSTy7k3lY7Ss74VnGlaPnrU%3D 2025-06-29 03:57:24,449 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-efc5c224]) (logid:3b66a780) ===START=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&id=564b4e1d-cbd0-431e-964d-5e15e592ca06&response=json&signature=c1laI4ClExqCugfgAsDqmJ2v7gI%3D 2025-06-29 03:57:24,454 DEBUG [c.c.a.ApiServer] (qtp1513608173-20:[ctx-efc5c224, ctx-3eca8450]) (logid:3b66a780) CIDRs from which account 'Account [{"accountName":"hgx","id":5,"uuid":"6b11f146-3810-4f53-a521-358f96b1a404"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2025-06-29 03:57:24,458 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-efc5c224, ctx-3eca8450]) (logid:3b66a780) Account for user id a3701253-d083-4d2a-880c-e19bdcf4ab2e is Root Admin or Domain Admin, all APIs are allowed. 2025-06-29 03:57:24,458 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1513608173-20:[ctx-efc5c224, ctx-3eca8450]) (logid:3b66a780) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2025-06-29 03:57:24,458 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1513608173-20:[ctx-efc5c224, ctx-3eca8450]) (logid:3b66a780) API rate limiting is disabled. We will not use ApiRateLimitService. 2025-06-29 03:57:24,481 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7bdb1320]) (logid:1b60575b) HA health check task is running... 2025-06-29 03:57:24,529 DEBUG [c.c.a.ApiServlet] (qtp1513608173-20:[ctx-efc5c224, ctx-3eca8450, ctx-67fb6504]) (logid:3b66a780) ===END=== 127.0.0.1 -- GET apiKey=OgWy09j5D7qyVGmbhOQCGM3y6-FqwdxLCjfZWrh3lIUXHdHrypqQtItUPYIoC_j_z2e3d_DoYEdC2ceEkSnu7w&command=listNetworks&id=564b4e1d-cbd0-431e-964d-5e15e592ca06&response=json&signature=c1laI4ClExqCugfgAsDqmJ2v7gI%3D 2025-06-29 03:57:25,375 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ee3252a]) (logid:7348e801) No inactive management server node found 2025-06-29 03:57:25,375 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ee3252a]) (logid:7348e801) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5348989]) (logid:60b50f86) No inactive management server node found 2025-06-29 03:57:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5348989]) (logid:60b50f86) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6ef17642]) (logid:e2cd064d) Enabled non-edge zones available for scan: 1 2025-06-29 03:57:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6ef17642]) (logid:e2cd064d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:57:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6ef17642]) (logid:e2cd064d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:57:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9c69b293]) (logid:43685381) Enabled non-edge zones available for scan: 1 2025-06-29 03:57:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9c69b293]) (logid:43685381) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:57:28,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f48041f]) (logid:329333ce) No inactive management server node found 2025-06-29 03:57:28,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f48041f]) (logid:329333ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 03:57:28,482 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-11bac6f9]) (logid:f23846ad) HA health check task is running... 2025-06-29 03:57:29,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62a4ace3]) (logid:a216fb4a) No inactive management server node found 2025-06-29 03:57:29,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62a4ace3]) (logid:a216fb4a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:30,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8396: Seq 7-8396: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:57:30,286 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8396 2025-06-29 03:57:31,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d73095f]) (logid:b33e43da) No inactive management server node found 2025-06-29 03:57:31,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d73095f]) (logid:b33e43da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:32,483 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9d482074]) (logid:df95a909) HA health check task is running... 2025-06-29 03:57:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14b696ae]) (logid:65f75a0b) No inactive management server node found 2025-06-29 03:57:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14b696ae]) (logid:65f75a0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33182323]) (logid:d4677451) No inactive management server node found 2025-06-29 03:57:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33182323]) (logid:d4677451) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce56278f]) (logid:d230f7c9) No inactive management server node found 2025-06-29 03:57:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce56278f]) (logid:d230f7c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:36,485 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f6e54198]) (logid:e1a4b127) HA health check task is running... 2025-06-29 03:57:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23b39d76]) (logid:a9305fb2) No inactive management server node found 2025-06-29 03:57:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23b39d76]) (logid:a9305fb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d3fc1e5]) (logid:b0e5b4aa) No inactive management server node found 2025-06-29 03:57:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d3fc1e5]) (logid:b0e5b4aa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:40,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8397: Seq 7-8397: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:57:40,286 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8397 2025-06-29 03:57:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29920564]) (logid:b2351819) No inactive management server node found 2025-06-29 03:57:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29920564]) (logid:b2351819) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:40,487 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-178914ff]) (logid:98d9f5df) HA health check task is running... 2025-06-29 03:57:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cbc6ee2]) (logid:944bf3b5) No inactive management server node found 2025-06-29 03:57:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cbc6ee2]) (logid:944bf3b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-f1c335be]) (logid:a5c75cbb) DbCollector is running... 2025-06-29 03:57:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5449807]) (logid:ed58503d) Found 0 routers to update status. 2025-06-29 03:57:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5449807]) (logid:ed58503d) Found 0 VPC's to update Redundant State. 2025-06-29 03:57:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5449807]) (logid:ed58503d) Found 0 networks to update RvR status. 2025-06-29 03:57:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83b7cbd6]) (logid:bbaa8ab8) No inactive management server node found 2025-06-29 03:57:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83b7cbd6]) (logid:bbaa8ab8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) ManagementServerCollector is running... 2025-06-29 03:57:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) DB properties were already loaded 2025-06-29 03:57:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:57:44,057 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Execution of process [4065709] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:57:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Exit value of process [4065709] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:57:44,058 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Process [4065709] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:57:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) usage install: null 2025-06-29 03:57:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Sessions active: 0 2025-06-29 03:57:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:57:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065712] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:57:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:57:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065717] for command [/bin/bash -c uname -r ]. 2025-06-29 03:57:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:57:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065718] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:57:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) system memory from /proc: 4887515136 2025-06-29 03:57:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:57:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065723] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:57:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) free memory from /proc: 272842752 2025-06-29 03:57:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:57:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065728] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:57:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) used memory from /proc: 1043152 2025-06-29 03:57:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:57:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065729] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:57:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:57:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065730] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:57:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:57:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065731] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:57:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:57:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065738] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:57:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:57:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Successfully executed process [4065739] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:57:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail buffersdirect.capacity, 318092 2025-06-29 03:57:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail buffersdirect.count, 90 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail buffersdirect.used, 318092 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail buffersmapped.capacity, 0 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail buffersmapped.count, 0 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail buffersmapped.used, 0 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail gcPS-Scavenge.count, 106 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail gcPS-Scavenge.time, 514 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail jvmuptime, 1323835 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memoryheap.committed, 536870912 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memoryheap.init, 77594624 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memoryheap.usage, 0.1874445093034945 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorynon-heap.committed, 193462272 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorynon-heap.max, -1 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorynon-heap.usage, -1.90005384E8 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorynon-heap.used, 190005384 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10264137896560917 2025-06-29 03:57:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20108328889925683 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.Compressed-Class-Space.usage, 0.01236734539270401 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.Metaspace.usage, 0.9933333672096736 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.PS-Eden-Space.usage, 0.23837475092235205 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorytotal.committed, 730333184 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorytotal.init, 85262336 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorytotal.max, 1908932607 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail memorytotal.used, 547824320 2025-06-29 03:57:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail threadsdeadlocks, [] 2025-06-29 03:57:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail threadsnew.count, 0 2025-06-29 03:57:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:57:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-b1f107ff]) (logid:9a962448) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":179774848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.19,"totalInit":85262336,"totalUsed":547092848,"totalCommitted":730333184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357818936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":272842752,"systemMemoryUsed":1043152,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.19,0.55,0.33],"systemCyclesUsage":[66403490,53826405,4589126458],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:57:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 43, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":179774848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.19,"totalInit":85262336,"totalUsed":547092848,"totalCommitted":730333184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357818936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":272842752,"systemMemoryUsed":1043152,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.19,0.55,0.33],"systemCyclesUsage":[66403490,53826405,4589126458],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:57:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:57:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-12:[ctx-ba1f5078]) (logid:72eda435) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":179774848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.19,"totalInit":85262336,"totalUsed":547092848,"totalCommitted":730333184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357818936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":272842752,"systemMemoryUsed":1043152,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.19,0.55,0.33],"systemCyclesUsage":[66403490,53826405,4589126458],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:57:44,144 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 03:57:44,144 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 43, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":179774848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.19,"totalInit":85262336,"totalUsed":547092848,"totalCommitted":730333184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357818936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":272842752,"systemMemoryUsed":1043152,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.19,0.55,0.33],"systemCyclesUsage":[66403490,53826405,4589126458],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:57:44,488 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5100d4c8]) (logid:7394a2f7) HA health check task is running... 2025-06-29 03:57:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f99ec29]) (logid:d60ee2dd) No inactive management server node found 2025-06-29 03:57:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f99ec29]) (logid:d60ee2dd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:57:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:57:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2628a55d]) (logid:0aa1e560) No inactive management server node found 2025-06-29 03:57:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2628a55d]) (logid:0aa1e560) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:57:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:57:47,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:47,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:47,091 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:57:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:57:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10fbaa00]) (logid:4e1574dd) No inactive management server node found 2025-06-29 03:57:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10fbaa00]) (logid:4e1574dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:48,067 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-fe929090]) (logid:199cdb44) ClusterDRS.poll is being called at 2025-06-29T01:58:00+0000 2025-06-29 03:57:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-fe929090]) (logid:199cdb44) Removed 0 old drs migration plans 2025-06-29 03:57:48,088 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7578c050]) (logid:d45b9c14) Resetting hosts suitable for reconnect 2025-06-29 03:57:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7578c050]) (logid:d45b9c14) Completed resetting hosts suitable for reconnect 2025-06-29 03:57:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7578c050]) (logid:d45b9c14) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:57:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7578c050]) (logid:d45b9c14) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:57:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7578c050]) (logid:d45b9c14) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:57:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7578c050]) (logid:d45b9c14) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:57:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-8dc4a931]) (logid:a41c3506) VM scheduler.poll is being called at 2025-06-29T01:58:00+0000 2025-06-29 03:57:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-8dc4a931]) (logid:a41c3506) Got 0 scheduled jobs to be executed at 2025-06-29T01:58:00+0000 2025-06-29 03:57:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-8dc4a931]) (logid:a41c3506) Cleaned up 0 VM scheduled job entries 2025-06-29 03:57:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-875bd419]) (logid:c211a918) VM Operation Thread Running 2025-06-29 03:57:48,489 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ae5bea6c]) (logid:2e249907) HA health check task is running... 2025-06-29 03:57:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-572edcaa]) (logid:2e349c99) No inactive management server node found 2025-06-29 03:57:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-572edcaa]) (logid:2e349c99) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:50,262 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8399: Seq 7-8399: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:57:50,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8399 2025-06-29 03:57:50,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a348886a]) (logid:d328af09) No inactive management server node found 2025-06-29 03:57:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a348886a]) (logid:d328af09) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:52,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e99538b9]) (logid:d17df205) No inactive management server node found 2025-06-29 03:57:52,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e99538b9]) (logid:d17df205) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:52,491 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ef8ac652]) (logid:70232f02) HA health check task is running... 2025-06-29 03:57:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4349796]) (logid:87ae620f) No inactive management server node found 2025-06-29 03:57:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4349796]) (logid:87ae620f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ba859d2]) (logid:43b79486) No inactive management server node found 2025-06-29 03:57:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ba859d2]) (logid:43b79486) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:56,492 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-718c0e18]) (logid:fdd7d95a) HA health check task is running... 2025-06-29 03:57:56,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6b2b7d6]) (logid:c1cc1423) No inactive management server node found 2025-06-29 03:57:56,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6b2b7d6]) (logid:c1cc1423) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9fe0b1ae]) (logid:e1a3e93a) Enabled non-edge zones available for scan: 1 2025-06-29 03:57:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9fe0b1ae]) (logid:e1a3e93a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:57:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9fe0b1ae]) (logid:e1a3e93a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:57:58,100 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-016a67b3]) (logid:8d3472b1) AutoScaling Monitor is running... 2025-06-29 03:57:58,106 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-e728e82a]) (logid:a72cb470) Starting to remove all volume_stats rows older than [Sat Jun 28 15:57:58 CEST 2025]. 2025-06-29 03:57:58,106 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-e728e82a]) (logid:a72cb470) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:57:58 CEST 2025]. 2025-06-29 03:57:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-036162ca]) (logid:13f76d67) Enabled non-edge zones available for scan: 1 2025-06-29 03:57:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-036162ca]) (logid:13f76d67) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:57:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0d3467d]) (logid:fcc0cbb3) No inactive management server node found 2025-06-29 03:57:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0d3467d]) (logid:fcc0cbb3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:58,694 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-dd278e5e]) (logid:97e43f93) Starting to remove all vm_stats rows older than [Sat Jun 28 15:57:58 CEST 2025]. 2025-06-29 03:57:58,729 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-dd278e5e]) (logid:97e43f93) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:57:58 CEST 2025]. 2025-06-29 03:57:59,846 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:57:59,847 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:57:59,849 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) Seq 2-4725683383995203679: Routed from 90520738888109 2025-06-29 03:57:59,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62387f7c]) (logid:2bf9ca30) No inactive management server node found 2025-06-29 03:57:59,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62387f7c]) (logid:2bf9ca30) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:57:59,893 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) Seq 1-4725683383995203679: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:57:59,895 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:57:59,895 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) Seq 2-3926857400090361964: Routed from 90520738888109 2025-06-29 03:57:59,940 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-dcd176e2]) (logid:7a8e39f7) Seq 2-3926857400090361964: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:58:00,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8400: Seq 7-8400: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:00,285 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8400 2025-06-29 03:58:00,494 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0f56ac14]) (logid:b16482aa) HA health check task is running... 2025-06-29 03:58:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e01fabf4]) (logid:5b5d2d7d) No inactive management server node found 2025-06-29 03:58:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e01fabf4]) (logid:5b5d2d7d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72232d7f]) (logid:4963c7a2) No inactive management server node found 2025-06-29 03:58:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72232d7f]) (logid:4963c7a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:03,779 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:58:03,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:58:03,784 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) Seq 2-4725683383995203680: Routed from 90520738888109 2025-06-29 03:58:03,837 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) Seq 1-4725683383995203680: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:58:03,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:58:03,884 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) Seq 2-3926857400090361965: Routed from 90520738888109 2025-06-29 03:58:03,973 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-62c11516]) (logid:2c8ab224) Seq 2-3926857400090361965: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:58:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccaa39a1]) (logid:97e9e9b1) No inactive management server node found 2025-06-29 03:58:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccaa39a1]) (logid:97e9e9b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:04,495 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3700f551]) (logid:9fe2643f) HA health check task is running... 2025-06-29 03:58:04,533 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) StorageCollector is running... 2025-06-29 03:58:04,540 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:58:04,540 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:58:04,540 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:58:04,541 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-3938116399158788125: Routed from 90520738888109 2025-06-29 03:58:04,597 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 4-3938116399158788125: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:58:04,601 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:58:04,601 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:58:04,601 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:58:04,602 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-3926857400090361966: Routed from 90520738888109 2025-06-29 03:58:04,613 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-3926857400090361966: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:58:04,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:58:04,624 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:58:04,624 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:58:04,625 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-4725683383995203681: Routed from 90520738888109 2025-06-29 03:58:04,673 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 1-4725683383995203681: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:58:04,684 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:58:04,684 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:58:04,684 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:58:04,685 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-4725683383995203682: Routed from 90520738888109 2025-06-29 03:58:04,770 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 1-4725683383995203682: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:58:04,771 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:58:04,772 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:58:04,772 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:58:04,773 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-3926857400090361967: Routed from 90520738888109 2025-06-29 03:58:04,817 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-af6564d7]) (logid:b06dc116) Seq 2-3926857400090361967: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:58:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcd19356]) (logid:857a2ca2) No inactive management server node found 2025-06-29 03:58:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcd19356]) (logid:857a2ca2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5cc0781]) (logid:8360cf09) No inactive management server node found 2025-06-29 03:58:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5cc0781]) (logid:8360cf09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:08,497 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cb793622]) (logid:a00f885f) HA health check task is running... 2025-06-29 03:58:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e16fff6f]) (logid:5b99d000) No inactive management server node found 2025-06-29 03:58:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e16fff6f]) (logid:5b99d000) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:10,263 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8401: Seq 7-8401: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:10,273 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8401 2025-06-29 03:58:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24603460]) (logid:ef2438bb) No inactive management server node found 2025-06-29 03:58:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24603460]) (logid:ef2438bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-634cd76d]) (logid:188ab2aa) No inactive management server node found 2025-06-29 03:58:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-634cd76d]) (logid:188ab2aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:12,498 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-058c7d64]) (logid:57862a3d) HA health check task is running... 2025-06-29 03:58:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-237c7459]) (logid:757c2f61) Found 0 routers to update status. 2025-06-29 03:58:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-237c7459]) (logid:757c2f61) Found 0 VPC's to update Redundant State. 2025-06-29 03:58:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-237c7459]) (logid:757c2f61) Found 0 networks to update RvR status. 2025-06-29 03:58:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d23b381]) (logid:631b9935) No inactive management server node found 2025-06-29 03:58:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d23b381]) (logid:631b9935) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:14,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3789c2dc]) (logid:9a89cc9a) No inactive management server node found 2025-06-29 03:58:14,869 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3789c2dc]) (logid:9a89cc9a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:16,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e39ed96c]) (logid:9270dc32) No inactive management server node found 2025-06-29 03:58:16,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e39ed96c]) (logid:9270dc32) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:16,499 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-010b3995]) (logid:56cb384a) HA health check task is running... 2025-06-29 03:58:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e7170af]) (logid:049ef809) No inactive management server node found 2025-06-29 03:58:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e7170af]) (logid:049ef809) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02cb40d8]) (logid:2ba675e9) No inactive management server node found 2025-06-29 03:58:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-02cb40d8]) (logid:2ba675e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:20,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8402: Seq 7-8402: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:20,286 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8402 2025-06-29 03:58:20,501 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-41c7b12b]) (logid:184870c8) HA health check task is running... 2025-06-29 03:58:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e2da044]) (logid:897312db) No inactive management server node found 2025-06-29 03:58:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e2da044]) (logid:897312db) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-866e968a]) (logid:e1f9d658) No inactive management server node found 2025-06-29 03:58:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-866e968a]) (logid:e1f9d658) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6a82f90]) (logid:1ecef21f) No inactive management server node found 2025-06-29 03:58:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6a82f90]) (logid:1ecef21f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:24,502 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b6457a66]) (logid:df2443cf) HA health check task is running... 2025-06-29 03:58:25,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5481a69d]) (logid:00e8fe30) No inactive management server node found 2025-06-29 03:58:25,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5481a69d]) (logid:00e8fe30) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0daa534a]) (logid:171e0458) No inactive management server node found 2025-06-29 03:58:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0daa534a]) (logid:171e0458) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c9941483]) (logid:f22111d4) Enabled non-edge zones available for scan: 1 2025-06-29 03:58:28,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c9941483]) (logid:f22111d4) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:58:28,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c9941483]) (logid:f22111d4) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:58:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7c68ced9]) (logid:b7a4e480) Enabled non-edge zones available for scan: 1 2025-06-29 03:58:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7c68ced9]) (logid:b7a4e480) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:58:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60d01914]) (logid:8604f967) No inactive management server node found 2025-06-29 03:58:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60d01914]) (logid:8604f967) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:28,504 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bce49a50]) (logid:c6c22bd4) HA health check task is running... 2025-06-29 03:58:29,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-516d46b1]) (logid:9f960aa4) No inactive management server node found 2025-06-29 03:58:29,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-516d46b1]) (logid:9f960aa4) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:30,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8403: Seq 7-8403: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:30,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8403 2025-06-29 03:58:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c424f677]) (logid:a8cf4690) No inactive management server node found 2025-06-29 03:58:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c424f677]) (logid:a8cf4690) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:32,505 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0e4c1c42]) (logid:bfff5262) HA health check task is running... 2025-06-29 03:58:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd8858e1]) (logid:9984a583) No inactive management server node found 2025-06-29 03:58:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd8858e1]) (logid:9984a583) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:34,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94764ce8]) (logid:312fb0c0) No inactive management server node found 2025-06-29 03:58:34,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94764ce8]) (logid:312fb0c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:35,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8404: Seq 7-8404: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:35,285 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8404 2025-06-29 03:58:35,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bb0de50]) (logid:23f8e942) No inactive management server node found 2025-06-29 03:58:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bb0de50]) (logid:23f8e942) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:36,507 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c9757035]) (logid:8d053dd2) HA health check task is running... 2025-06-29 03:58:37,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-840a3811]) (logid:0e770b01) No inactive management server node found 2025-06-29 03:58:37,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-840a3811]) (logid:0e770b01) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:38,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c609c33f]) (logid:4044b989) No inactive management server node found 2025-06-29 03:58:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c609c33f]) (logid:4044b989) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:40,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84a4829f]) (logid:71c0d09c) No inactive management server node found 2025-06-29 03:58:40,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84a4829f]) (logid:71c0d09c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:40,509 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a58e2f93]) (logid:9b5a6834) HA health check task is running... 2025-06-29 03:58:41,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dd607e3]) (logid:d9f45ca9) No inactive management server node found 2025-06-29 03:58:41,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dd607e3]) (logid:d9f45ca9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-717057ac]) (logid:d180682a) DbCollector is running... 2025-06-29 03:58:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b923e9d1]) (logid:b3450cfe) Found 0 routers to update status. 2025-06-29 03:58:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b923e9d1]) (logid:b3450cfe) Found 0 VPC's to update Redundant State. 2025-06-29 03:58:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b923e9d1]) (logid:b3450cfe) Found 0 networks to update RvR status. 2025-06-29 03:58:43,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df68e93f]) (logid:78a4850f) No inactive management server node found 2025-06-29 03:58:43,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df68e93f]) (logid:78a4850f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) ManagementServerCollector is running... 2025-06-29 03:58:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) DB properties were already loaded 2025-06-29 03:58:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:58:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Execution of process [4065752] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:58:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Exit value of process [4065752] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:58:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Process [4065752] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:58:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) usage install: null 2025-06-29 03:58:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Sessions active: 0 2025-06-29 03:58:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:58:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065755] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:58:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:58:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065760] for command [/bin/bash -c uname -r ]. 2025-06-29 03:58:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:58:44,089 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065761] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:58:44,089 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) system memory from /proc: 4887515136 2025-06-29 03:58:44,089 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:58:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065766] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:58:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) free memory from /proc: 305156096 2025-06-29 03:58:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:58:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065771] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:58:44,109 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) used memory from /proc: 1043408 2025-06-29 03:58:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:58:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065772] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:58:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:58:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065773] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:58:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:58:44,133 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065774] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:58:44,133 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:58:44,137 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065781] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:58:44,137 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:58:44,142 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Successfully executed process [4065782] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:58:44,144 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail buffersdirect.capacity, 318092 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail buffersdirect.count, 90 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail buffersdirect.used, 318092 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail buffersmapped.capacity, 0 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail buffersmapped.count, 0 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail buffersmapped.used, 0 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail gcPS-Scavenge.count, 106 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail gcPS-Scavenge.time, 514 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail jvmuptime, 1383848 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memoryheap.committed, 536870912 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memoryheap.init, 77594624 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memoryheap.usage, 0.19843889638245416 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorynon-heap.committed, 193789952 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorynon-heap.max, -1 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorynon-heap.usage, -1.90477072E8 2025-06-29 03:58:44,145 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorynon-heap.used, 190477072 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10466167355371901 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2024902522744693 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.Compressed-Class-Space.usage, 0.012368284165859222 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.Metaspace.usage, 0.9932203360483156 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.PS-Eden-Space.usage, 0.2704754919404552 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorytotal.committed, 730660864 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorytotal.init, 85262336 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorytotal.max, 1908932607 2025-06-29 03:58:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail memorytotal.used, 569518680 2025-06-29 03:58:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail threadsdeadlocks, [] 2025-06-29 03:58:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail threadsnew.count, 0 2025-06-29 03:58:44,148 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) not storing detail threadstimed_waiting.count, 82 2025-06-29 03:58:44,149 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-cb6ee300]) (logid:e25c6e7a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":158575728,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1383760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.52,"totalInit":85262336,"totalUsed":568658232,"totalCommitted":730595328,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":378806480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":305156096,"systemMemoryUsed":1043408,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.52,0.77,0.42],"systemCyclesUsage":[66403951,53826726,4589149413],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:58:44,150 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 45, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":158575728,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1383760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.52,"totalInit":85262336,"totalUsed":568658232,"totalCommitted":730595328,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":378806480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":305156096,"systemMemoryUsed":1043408,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.52,0.77,0.42],"systemCyclesUsage":[66403951,53826726,4589149413],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:58:44,150 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:58:44,158 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-13:[ctx-cc515f10]) (logid:27fb6e32) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":158575728,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1383760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.52,"totalInit":85262336,"totalUsed":568658232,"totalCommitted":730595328,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":378806480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":305156096,"systemMemoryUsed":1043408,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.52,0.77,0.42],"systemCyclesUsage":[66403951,53826726,4589149413],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:58:44,159 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 03:58:44,159 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 9ms. agent: 0, pdu seq: 45, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":158575728,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1383760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.52,"totalInit":85262336,"totalUsed":568658232,"totalCommitted":730595328,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":378806480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":198,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":305156096,"systemMemoryUsed":1043408,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.52,0.77,0.42],"systemCyclesUsage":[66403951,53826726,4589149413],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:58:44,510 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c6ba2b79]) (logid:59acf87b) HA health check task is running... 2025-06-29 03:58:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e889aa9]) (logid:e8c5c7ac) No inactive management server node found 2025-06-29 03:58:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e889aa9]) (logid:e8c5c7ac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:45,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8405: Seq 7-8405: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:45,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8405 2025-06-29 03:58:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:58:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:58:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4440bb9]) (logid:cf426630) No inactive management server node found 2025-06-29 03:58:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4440bb9]) (logid:cf426630) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:58:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:58:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:58:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:58:47,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:58:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:58:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:58:47,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:58:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4338a54]) (logid:5a52ad68) No inactive management server node found 2025-06-29 03:58:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4338a54]) (logid:5a52ad68) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-01d08def]) (logid:43045a27) ClusterDRS.poll is being called at 2025-06-29T01:59:00+0000 2025-06-29 03:58:48,074 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-01d08def]) (logid:43045a27) Removed 0 old drs migration plans 2025-06-29 03:58:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-60e541dc]) (logid:a994e53d) VM scheduler.poll is being called at 2025-06-29T01:59:00+0000 2025-06-29 03:58:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-60e541dc]) (logid:a994e53d) Got 0 scheduled jobs to be executed at 2025-06-29T01:59:00+0000 2025-06-29 03:58:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-60e541dc]) (logid:a994e53d) Cleaned up 0 VM scheduled job entries 2025-06-29 03:58:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ae0c11ac]) (logid:e264bd18) VM Operation Thread Running 2025-06-29 03:58:48,511 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e2954f1d]) (logid:1a0a2cce) HA health check task is running... 2025-06-29 03:58:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6fa94a8]) (logid:9458b328) No inactive management server node found 2025-06-29 03:58:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6fa94a8]) (logid:9458b328) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:50,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09e2faad]) (logid:332462d0) No inactive management server node found 2025-06-29 03:58:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09e2faad]) (logid:332462d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-790a0120]) (logid:7d409d3f) No inactive management server node found 2025-06-29 03:58:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-790a0120]) (logid:7d409d3f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:52,513 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-806644d7]) (logid:5b781c55) HA health check task is running... 2025-06-29 03:58:53,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2f1951]) (logid:415c3a67) No inactive management server node found 2025-06-29 03:58:53,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2f1951]) (logid:415c3a67) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:55,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8407: Seq 7-8407: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:58:55,288 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8407 2025-06-29 03:58:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cb95ecf]) (logid:aa8c09b0) No inactive management server node found 2025-06-29 03:58:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cb95ecf]) (logid:aa8c09b0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:56,514 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-61cb1687]) (logid:f1c51268) HA health check task is running... 2025-06-29 03:58:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ff463a9]) (logid:fd4aa3bc) No inactive management server node found 2025-06-29 03:58:56,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ff463a9]) (logid:fd4aa3bc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0921b3b5]) (logid:eb52fcdb) Enabled non-edge zones available for scan: 1 2025-06-29 03:58:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0921b3b5]) (logid:eb52fcdb) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:58:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0921b3b5]) (logid:eb52fcdb) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:58:58,101 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-ad91e08c]) (logid:f21f55c0) AutoScaling Monitor is running... 2025-06-29 03:58:58,108 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-86caceab]) (logid:d492f681) Starting to remove all volume_stats rows older than [Sat Jun 28 15:58:58 CEST 2025]. 2025-06-29 03:58:58,109 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-86caceab]) (logid:d492f681) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:58:58 CEST 2025]. 2025-06-29 03:58:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1e3f0362]) (logid:5ff84801) Enabled non-edge zones available for scan: 1 2025-06-29 03:58:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1e3f0362]) (logid:5ff84801) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:58:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ef6ab94]) (logid:1c4be68e) No inactive management server node found 2025-06-29 03:58:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ef6ab94]) (logid:1c4be68e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:58,730 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-534bee0b]) (logid:5b7622e6) Starting to remove all vm_stats rows older than [Sat Jun 28 15:58:58 CEST 2025]. 2025-06-29 03:58:58,765 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-534bee0b]) (logid:5b7622e6) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:58:58 CEST 2025]. 2025-06-29 03:58:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc2d4f35]) (logid:13663622) No inactive management server node found 2025-06-29 03:58:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc2d4f35]) (logid:13663622) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:58:59,943 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) HostStatsCollector is running to process 2 UP hosts 2025-06-29 03:58:59,945 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:58:59,947 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) Seq 2-4725683383995203683: Routed from 90520738888109 2025-06-29 03:58:59,991 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) Seq 1-4725683383995203683: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:58:59,992 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 03:58:59,993 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) Seq 2-3926857400090361968: Routed from 90520738888109 2025-06-29 03:59:00,038 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-843b29ba]) (logid:2a783567) Seq 2-3926857400090361968: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 03:59:00,516 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-faf7090d]) (logid:32485dc7) HA health check task is running... 2025-06-29 03:59:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0d18e3a]) (logid:3253d68b) No inactive management server node found 2025-06-29 03:59:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0d18e3a]) (logid:3253d68b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c056243e]) (logid:b8b5fb87) No inactive management server node found 2025-06-29 03:59:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c056243e]) (logid:b8b5fb87) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:04,036 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 03:59:04,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:59:04,041 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) Seq 2-4725683383995203684: Routed from 90520738888109 2025-06-29 03:59:04,098 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) Seq 1-4725683383995203684: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:59:04,147 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 03:59:04,148 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) Seq 2-3926857400090361969: Routed from 90520738888109 2025-06-29 03:59:04,239 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-ce59a3f2]) (logid:bfa8ad1e) Seq 2-3926857400090361969: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 03:59:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f051ccd1]) (logid:ad77647d) No inactive management server node found 2025-06-29 03:59:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f051ccd1]) (logid:ad77647d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:04,517 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7453edf4]) (logid:d4712cdd) HA health check task is running... 2025-06-29 03:59:04,818 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) StorageCollector is running... 2025-06-29 03:59:04,824 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:59:04,824 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:59:04,824 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:59:04,825 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 2-3938116399158788126: Routed from 90520738888109 2025-06-29 03:59:04,902 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 4-3938116399158788126: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:59:04,908 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:59:04,908 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:59:04,908 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:59:04,909 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 2-4725683383995203685: Routed from 90520738888109 2025-06-29 03:59:04,957 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 1-4725683383995203685: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:59:04,968 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:59:04,968 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:59:04,968 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:59:04,969 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 2-4725683383995203686: Routed from 90520738888109 2025-06-29 03:59:05,077 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 1-4725683383995203686: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:59:05,087 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:59:05,087 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:59:05,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:59:05,088 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 2-4725683383995203687: Routed from 90520738888109 2025-06-29 03:59:05,172 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 1-4725683383995203687: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:59:05,174 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 03:59:05,174 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 03:59:05,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 03:59:05,175 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 2-3926857400090361970: Routed from 90520738888109 2025-06-29 03:59:05,179 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5bdb1dc6]) (logid:038458aa) Seq 2-3926857400090361970: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 03:59:05,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8408: Seq 7-8408: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:59:05,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8408 2025-06-29 03:59:05,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1736a472]) (logid:3eac1396) No inactive management server node found 2025-06-29 03:59:05,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1736a472]) (logid:3eac1396) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bc176ba]) (logid:2d01851d) No inactive management server node found 2025-06-29 03:59:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bc176ba]) (logid:2d01851d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:08,518 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-feb32c92]) (logid:2c96fb2e) HA health check task is running... 2025-06-29 03:59:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17dbdf49]) (logid:f31b32ea) No inactive management server node found 2025-06-29 03:59:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17dbdf49]) (logid:f31b32ea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65899d41]) (logid:8115e530) No inactive management server node found 2025-06-29 03:59:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65899d41]) (logid:8115e530) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:11,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97aeb5ab]) (logid:435d4a70) No inactive management server node found 2025-06-29 03:59:11,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97aeb5ab]) (logid:435d4a70) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:12,520 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d1243739]) (logid:e50257f4) HA health check task is running... 2025-06-29 03:59:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6bb7194f]) (logid:251bfaa2) Found 0 routers to update status. 2025-06-29 03:59:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6bb7194f]) (logid:251bfaa2) Found 0 VPC's to update Redundant State. 2025-06-29 03:59:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6bb7194f]) (logid:251bfaa2) Found 0 networks to update RvR status. 2025-06-29 03:59:13,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f8b47fe]) (logid:80779df5) No inactive management server node found 2025-06-29 03:59:13,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f8b47fe]) (logid:80779df5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3821173]) (logid:8dec73db) No inactive management server node found 2025-06-29 03:59:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3821173]) (logid:8dec73db) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:15,266 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8409: Seq 7-8409: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:59:15,288 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8409 2025-06-29 03:59:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c527ffd]) (logid:4f20f73f) No inactive management server node found 2025-06-29 03:59:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c527ffd]) (logid:4f20f73f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:16,521 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-772aa90f]) (logid:6c9d8743) HA health check task is running... 2025-06-29 03:59:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-686e6bef]) (logid:623334c8) No inactive management server node found 2025-06-29 03:59:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-686e6bef]) (logid:623334c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:18,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2bba2b64]) (logid:efcf1226) Resetting hosts suitable for reconnect 2025-06-29 03:59:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2bba2b64]) (logid:efcf1226) Completed resetting hosts suitable for reconnect 2025-06-29 03:59:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2bba2b64]) (logid:efcf1226) Acquiring hosts for clusters already owned by this management server 2025-06-29 03:59:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2bba2b64]) (logid:efcf1226) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 03:59:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2bba2b64]) (logid:efcf1226) Acquiring hosts for clusters not owned by any management server 2025-06-29 03:59:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2bba2b64]) (logid:efcf1226) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 03:59:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66f80ac4]) (logid:04b75a14) No inactive management server node found 2025-06-29 03:59:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66f80ac4]) (logid:04b75a14) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:20,523 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cfe80b85]) (logid:b4ffe972) HA health check task is running... 2025-06-29 03:59:20,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7703f27]) (logid:27c911a6) No inactive management server node found 2025-06-29 03:59:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7703f27]) (logid:27c911a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:22,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-607579e9]) (logid:1374c2c7) No inactive management server node found 2025-06-29 03:59:22,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-607579e9]) (logid:1374c2c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8b91346]) (logid:64a1204d) No inactive management server node found 2025-06-29 03:59:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8b91346]) (logid:64a1204d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:24,524 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b76f6112]) (logid:5332a210) HA health check task is running... 2025-06-29 03:59:25,267 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8410: Seq 7-8410: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:59:25,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8410 2025-06-29 03:59:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f2ab9e3]) (logid:818d1696) No inactive management server node found 2025-06-29 03:59:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f2ab9e3]) (logid:818d1696) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81356230]) (logid:c6428206) No inactive management server node found 2025-06-29 03:59:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81356230]) (logid:c6428206) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-536f85cf]) (logid:febd9bd8) Enabled non-edge zones available for scan: 1 2025-06-29 03:59:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-536f85cf]) (logid:febd9bd8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:59:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-536f85cf]) (logid:febd9bd8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:59:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0312f57e]) (logid:f5ed43cf) Enabled non-edge zones available for scan: 1 2025-06-29 03:59:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0312f57e]) (logid:f5ed43cf) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:59:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-730b46b3]) (logid:f03079a0) No inactive management server node found 2025-06-29 03:59:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-730b46b3]) (logid:f03079a0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:28,526 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-66f9fedf]) (logid:cece379f) HA health check task is running... 2025-06-29 03:59:29,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fbf3b5f]) (logid:aa3bf60d) No inactive management server node found 2025-06-29 03:59:29,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fbf3b5f]) (logid:aa3bf60d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:31,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2188b29f]) (logid:d22e405b) No inactive management server node found 2025-06-29 03:59:31,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2188b29f]) (logid:d22e405b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:32,527 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8baf5227]) (logid:84f71598) HA health check task is running... 2025-06-29 03:59:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-525c0769]) (logid:fee91343) No inactive management server node found 2025-06-29 03:59:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-525c0769]) (logid:fee91343) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:34,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fc680b2]) (logid:16836ab0) No inactive management server node found 2025-06-29 03:59:34,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fc680b2]) (logid:16836ab0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:35,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8411: Seq 7-8411: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:59:35,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8411 2025-06-29 03:59:35,874 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa58ac66]) (logid:e632eed2) No inactive management server node found 2025-06-29 03:59:35,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa58ac66]) (logid:e632eed2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:36,529 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f7cae961]) (logid:1bcea12c) HA health check task is running... 2025-06-29 03:59:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81920edd]) (logid:2f950f22) No inactive management server node found 2025-06-29 03:59:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81920edd]) (logid:2f950f22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c1e0a99]) (logid:0ab62851) No inactive management server node found 2025-06-29 03:59:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c1e0a99]) (logid:0ab62851) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:40,267 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8412: Seq 7-8412: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:59:40,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8412 2025-06-29 03:59:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a060583]) (logid:25626543) No inactive management server node found 2025-06-29 03:59:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a060583]) (logid:25626543) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:40,530 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-96eba262]) (logid:29573a7e) HA health check task is running... 2025-06-29 03:59:41,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a53f611]) (logid:3ee41bb8) No inactive management server node found 2025-06-29 03:59:41,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a53f611]) (logid:3ee41bb8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-5597520b]) (logid:aa560b19) DbCollector is running... 2025-06-29 03:59:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f0c9ef6f]) (logid:6f6f2928) Found 0 routers to update status. 2025-06-29 03:59:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f0c9ef6f]) (logid:6f6f2928) Found 0 VPC's to update Redundant State. 2025-06-29 03:59:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f0c9ef6f]) (logid:6f6f2928) Found 0 networks to update RvR status. 2025-06-29 03:59:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-970e7353]) (logid:83c0b203) No inactive management server node found 2025-06-29 03:59:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-970e7353]) (logid:83c0b203) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) ManagementServerCollector is running... 2025-06-29 03:59:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) DB properties were already loaded 2025-06-29 03:59:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 03:59:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Execution of process [4065789] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 03:59:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Exit value of process [4065789] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 03:59:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Process [4065789] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 03:59:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) usage install: null 2025-06-29 03:59:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Sessions active: 0 2025-06-29 03:59:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:59:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065792] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 03:59:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c uname -r ]. 2025-06-29 03:59:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065797] for command [/bin/bash -c uname -r ]. 2025-06-29 03:59:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:59:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065798] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:59:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) system memory from /proc: 4887515136 2025-06-29 03:59:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:59:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065803] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 03:59:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) free memory from /proc: 308994048 2025-06-29 03:59:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:59:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065808] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 03:59:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) used memory from /proc: 1043664 2025-06-29 03:59:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 03:59:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065809] for command [/bin/bash -c uptime -s ]. 2025-06-29 03:59:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:59:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065810] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 03:59:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:59:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065811] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 03:59:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:59:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065818] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 03:59:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:59:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Successfully executed process [4065819] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail buffersdirect.capacity, 318092 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail buffersdirect.count, 90 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail buffersdirect.used, 318092 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail buffersmapped.capacity, 0 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail buffersmapped.count, 0 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail buffersmapped.used, 0 2025-06-29 03:59:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail gcPS-Scavenge.count, 106 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail gcPS-Scavenge.time, 514 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail jvmuptime, 1443829 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memoryheap.committed, 536870912 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memoryheap.init, 77594624 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memoryheap.usage, 0.21057250859219437 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorynon-heap.committed, 193855488 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorynon-heap.init, 7667712 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorynon-heap.max, -1 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorynon-heap.usage, -1.90591576E8 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorynon-heap.used, 190591576 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10511988469741403 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20267041856900056 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.Compressed-Class-Space.usage, 0.012369312345981598 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.Metaspace.usage, 0.9934723838283097 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.PS-Eden-Space.usage, 0.3051543269811595 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorytotal.committed, 730726400 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorytotal.init, 85262336 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorytotal.max, 1908932607 2025-06-29 03:59:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail memorytotal.used, 592560304 2025-06-29 03:59:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail threadsdeadlocks, [] 2025-06-29 03:59:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail threadsnew.count, 0 2025-06-29 03:59:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) not storing detail threadstimed_waiting.count, 83 2025-06-29 03:59:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-cc3b7560]) (logid:1cd4d90e) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":135773736,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.2,"totalInit":85262336,"totalUsed":591680144,"totalCommitted":730726400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":401968728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":308994048,"systemMemoryUsed":1043664,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.2,0.83,0.46],"systemCyclesUsage":[66404306,53827037,4589172487],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 03:59:44,130 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 47, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":135773736,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.2,"totalInit":85262336,"totalUsed":591680144,"totalCommitted":730726400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":401968728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":308994048,"systemMemoryUsed":1043664,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.2,0.83,0.46],"systemCyclesUsage":[66404306,53827037,4589172487],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:59:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 03:59:44,137 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-13:[ctx-5fa33db3]) (logid:1576cc7f) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":135773736,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.2,"totalInit":85262336,"totalUsed":591680144,"totalCommitted":730726400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":401968728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":308994048,"systemMemoryUsed":1043664,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.2,0.83,0.46],"systemCyclesUsage":[66404306,53827037,4589172487],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:59:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 03:59:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 47, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 3:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":135773736,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.2,"totalInit":85262336,"totalUsed":591680144,"totalCommitted":730726400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":401968728,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":199,"threadsWaitingCount":107,"systemMemoryTotal":4887515136,"systemMemoryFree":308994048,"systemMemoryUsed":1043664,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.2,0.83,0.46],"systemCyclesUsage":[66404306,53827037,4589172487],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 03:59:44,532 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d8f602da]) (logid:8a8b438c) HA health check task is running... 2025-06-29 03:59:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8128c340]) (logid:dd8bb26c) No inactive management server node found 2025-06-29 03:59:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8128c340]) (logid:dd8bb26c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 03:59:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 03:59:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99fbb659]) (logid:039fd955) No inactive management server node found 2025-06-29 03:59:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99fbb659]) (logid:039fd955) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:59:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:59:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 03:59:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 03:59:47,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:59:47,105 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:59:47,106 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 03:59:47,109 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 03:59:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-851dc1e3]) (logid:07be1d85) No inactive management server node found 2025-06-29 03:59:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-851dc1e3]) (logid:07be1d85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-b374f74a]) (logid:1fc9fc8c) ClusterDRS.poll is being called at 2025-06-29T02:00:00+0000 2025-06-29 03:59:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-b374f74a]) (logid:1fc9fc8c) Removed 0 old drs migration plans 2025-06-29 03:59:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0541b1bd]) (logid:11cb9a3c) VM scheduler.poll is being called at 2025-06-29T02:00:00+0000 2025-06-29 03:59:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0541b1bd]) (logid:11cb9a3c) Got 0 scheduled jobs to be executed at 2025-06-29T02:00:00+0000 2025-06-29 03:59:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0541b1bd]) (logid:11cb9a3c) Cleaned up 0 VM scheduled job entries 2025-06-29 03:59:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-42a41f24]) (logid:2e790965) VM Operation Thread Running 2025-06-29 03:59:48,533 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d4301243]) (logid:84b3cd92) HA health check task is running... 2025-06-29 03:59:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1ebfc33]) (logid:147fdc7e) No inactive management server node found 2025-06-29 03:59:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1ebfc33]) (logid:147fdc7e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:50,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8414: Seq 7-8414: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 03:59:50,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8414 2025-06-29 03:59:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-420dba99]) (logid:177e67c7) No inactive management server node found 2025-06-29 03:59:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-420dba99]) (logid:177e67c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3eb6e86a]) (logid:6850c0a6) No inactive management server node found 2025-06-29 03:59:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3eb6e86a]) (logid:6850c0a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:52,534 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f0b805b8]) (logid:a36cb2d0) HA health check task is running... 2025-06-29 03:59:53,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65687204]) (logid:35707c3e) No inactive management server node found 2025-06-29 03:59:53,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65687204]) (logid:35707c3e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:55,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d960a6c4]) (logid:bd6d1231) No inactive management server node found 2025-06-29 03:59:55,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d960a6c4]) (logid:bd6d1231) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:56,535 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-96cbe42a]) (logid:69f72760) HA health check task is running... 2025-06-29 03:59:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-855bd232]) (logid:146974de) No inactive management server node found 2025-06-29 03:59:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-855bd232]) (logid:146974de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1b9d208d]) (logid:c4957bde) Enabled non-edge zones available for scan: 1 2025-06-29 03:59:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1b9d208d]) (logid:c4957bde) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 03:59:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1b9d208d]) (logid:c4957bde) Zone [1] is ready to launch secondary storage VM. 2025-06-29 03:59:58,104 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-a8bca840]) (logid:80d9ef0d) AutoScaling Monitor is running... 2025-06-29 03:59:58,111 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-b98fcc5d]) (logid:58f4e70c) Starting to remove all volume_stats rows older than [Sat Jun 28 15:59:58 CEST 2025]. 2025-06-29 03:59:58,112 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-b98fcc5d]) (logid:58f4e70c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 15:59:58 CEST 2025]. 2025-06-29 03:59:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2a808027]) (logid:c68f9cac) Enabled non-edge zones available for scan: 1 2025-06-29 03:59:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2a808027]) (logid:c68f9cac) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 03:59:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-009b027a]) (logid:cf11f362) No inactive management server node found 2025-06-29 03:59:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-009b027a]) (logid:cf11f362) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 03:59:58,767 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-e3f20f13]) (logid:b15f6d2a) Starting to remove all vm_stats rows older than [Sat Jun 28 15:59:58 CEST 2025]. 2025-06-29 03:59:58,800 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-e3f20f13]) (logid:b15f6d2a) Removed a total of [8] vm_stats rows older than [Sat Jun 28 15:59:58 CEST 2025]. 2025-06-29 03:59:59,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9320caa6]) (logid:8148a8e6) No inactive management server node found 2025-06-29 03:59:59,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9320caa6]) (logid:8148a8e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:00,041 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:00:00,043 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:00:00,044 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) Seq 2-4725683383995203688: Routed from 90520738888109 2025-06-29 04:00:00,087 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) Seq 1-4725683383995203688: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:00:00,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:00:00,090 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) Seq 2-3926857400090361971: Routed from 90520738888109 2025-06-29 04:00:00,134 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b5a33322]) (logid:35142a24) Seq 2-3926857400090361971: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:00:00,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8415: Seq 7-8415: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:00:00,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8415 2025-06-29 04:00:00,537 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-05e910ff]) (logid:9841ca05) HA health check task is running... 2025-06-29 04:00:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cac13cd]) (logid:8019578e) No inactive management server node found 2025-06-29 04:00:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cac13cd]) (logid:8019578e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:00:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f74010e0]) (logid:4c770f17) No inactive management server node found 2025-06-29 04:00:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f74010e0]) (logid:4c770f17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:04,301 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:00:04,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:00:04,307 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) Seq 2-4725683383995203689: Routed from 90520738888109 2025-06-29 04:00:04,366 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) Seq 1-4725683383995203689: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:00:04,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b18c5e2f]) (logid:55855443) No inactive management server node found 2025-06-29 04:00:04,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b18c5e2f]) (logid:55855443) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:04,402 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:00:04,403 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) Seq 2-3926857400090361972: Routed from 90520738888109 2025-06-29 04:00:04,494 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-9d6a080a]) (logid:44a7c7aa) Seq 2-3926857400090361972: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:00:04,538 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-dbc16b8d]) (logid:580748fb) HA health check task is running... 2025-06-29 04:00:05,179 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) StorageCollector is running... 2025-06-29 04:00:05,186 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:00:05,186 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:00:05,186 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:00:05,187 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 2-3938116399158788127: Routed from 90520738888109 2025-06-29 04:00:05,244 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 4-3938116399158788127: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:00:05,248 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:00:05,248 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:00:05,248 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:00:05,250 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 2-4725683383995203690: Routed from 90520738888109 2025-06-29 04:00:05,302 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 1-4725683383995203690: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:00:05,326 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:00:05,326 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:00:05,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:00:05,328 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 2-4725683383995203691: Routed from 90520738888109 2025-06-29 04:00:05,417 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 1-4725683383995203691: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:00:05,428 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:00:05,428 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:00:05,428 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:00:05,429 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 2-4725683383995203692: Routed from 90520738888109 2025-06-29 04:00:05,514 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 1-4725683383995203692: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:00:05,516 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:00:05,516 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:00:05,516 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:00:05,517 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 2-3926857400090361973: Routed from 90520738888109 2025-06-29 04:00:05,521 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e38a2fc2]) (logid:e4aefa64) Seq 2-3926857400090361973: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:00:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d83af3a]) (logid:6f59e3b9) No inactive management server node found 2025-06-29 04:00:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d83af3a]) (logid:6f59e3b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ad9c4e4]) (logid:26adfedc) No inactive management server node found 2025-06-29 04:00:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ad9c4e4]) (logid:26adfedc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:08,539 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-54a027e0]) (logid:243a4d6b) HA health check task is running... 2025-06-29 04:00:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28827e07]) (logid:4a290c55) No inactive management server node found 2025-06-29 04:00:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28827e07]) (logid:4a290c55) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:10,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8416: Seq 7-8416: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:00:10,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8416 2025-06-29 04:00:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe8cf116]) (logid:dffafed5) No inactive management server node found 2025-06-29 04:00:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe8cf116]) (logid:dffafed5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0d0cd44]) (logid:1ec2f67a) No inactive management server node found 2025-06-29 04:00:11,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0d0cd44]) (logid:1ec2f67a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:12,541 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f3334381]) (logid:bcf387c7) HA health check task is running... 2025-06-29 04:00:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-df0451fc]) (logid:e085bf0b) Found 0 routers to update status. 2025-06-29 04:00:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-df0451fc]) (logid:e085bf0b) Found 0 VPC's to update Redundant State. 2025-06-29 04:00:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-df0451fc]) (logid:e085bf0b) Found 0 networks to update RvR status. 2025-06-29 04:00:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8855f527]) (logid:2db69205) No inactive management server node found 2025-06-29 04:00:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8855f527]) (logid:2db69205) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edfdb0da]) (logid:7b12f229) No inactive management server node found 2025-06-29 04:00:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edfdb0da]) (logid:7b12f229) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e73879f]) (logid:c8a2588d) No inactive management server node found 2025-06-29 04:00:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e73879f]) (logid:c8a2588d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:16,542 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-08386d9b]) (logid:4ae16f33) HA health check task is running... 2025-06-29 04:00:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bcb85c0]) (logid:07d32d4b) No inactive management server node found 2025-06-29 04:00:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bcb85c0]) (logid:07d32d4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a41311d]) (logid:fccb2eab) No inactive management server node found 2025-06-29 04:00:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a41311d]) (logid:fccb2eab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:20,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8417: Seq 7-8417: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:00:20,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8417 2025-06-29 04:00:20,544 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a78571ed]) (logid:54b95f0e) HA health check task is running... 2025-06-29 04:00:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f581fae2]) (logid:c17b2cbd) No inactive management server node found 2025-06-29 04:00:20,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f581fae2]) (logid:c17b2cbd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a66b642]) (logid:ce6f1ea2) No inactive management server node found 2025-06-29 04:00:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a66b642]) (logid:ce6f1ea2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-361a68f7]) (logid:7ddee685) No inactive management server node found 2025-06-29 04:00:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-361a68f7]) (logid:7ddee685) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:24,545 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-80195907]) (logid:a5afcacd) HA health check task is running... 2025-06-29 04:00:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbad90b3]) (logid:db5e4448) No inactive management server node found 2025-06-29 04:00:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbad90b3]) (logid:db5e4448) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-869b15f8]) (logid:68a67aa1) No inactive management server node found 2025-06-29 04:00:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-869b15f8]) (logid:68a67aa1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba094eee]) (logid:031f82c9) Enabled non-edge zones available for scan: 1 2025-06-29 04:00:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ba094eee]) (logid:031f82c9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:00:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ba094eee]) (logid:031f82c9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:00:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3b5523c8]) (logid:857ef778) Enabled non-edge zones available for scan: 1 2025-06-29 04:00:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3b5523c8]) (logid:857ef778) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:00:28,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12b67198]) (logid:50156359) No inactive management server node found 2025-06-29 04:00:28,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12b67198]) (logid:50156359) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:28,546 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c11313cc]) (logid:51018b90) HA health check task is running... 2025-06-29 04:00:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c63aeba0]) (logid:bcc97d99) No inactive management server node found 2025-06-29 04:00:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c63aeba0]) (logid:bcc97d99) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:30,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8418: Seq 7-8418: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:00:30,290 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8418 2025-06-29 04:00:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ba4364f]) (logid:61b37853) No inactive management server node found 2025-06-29 04:00:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ba4364f]) (logid:61b37853) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:32,547 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-44656027]) (logid:1f6fc302) HA health check task is running... 2025-06-29 04:00:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2716a9f3]) (logid:b817e3fc) No inactive management server node found 2025-06-29 04:00:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2716a9f3]) (logid:b817e3fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:34,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f072c155]) (logid:f1ddf9eb) No inactive management server node found 2025-06-29 04:00:34,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f072c155]) (logid:f1ddf9eb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4e67f8d]) (logid:4ea3f123) No inactive management server node found 2025-06-29 04:00:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4e67f8d]) (logid:4ea3f123) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:36,548 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-92970243]) (logid:7d228763) HA health check task is running... 2025-06-29 04:00:37,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a037215c]) (logid:ae27e0c2) No inactive management server node found 2025-06-29 04:00:37,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a037215c]) (logid:ae27e0c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-450d6b25]) (logid:bb0b30ca) No inactive management server node found 2025-06-29 04:00:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-450d6b25]) (logid:bb0b30ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:40,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8419: Seq 7-8419: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:00:40,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8419 2025-06-29 04:00:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0680c6e7]) (logid:1e6e3d66) No inactive management server node found 2025-06-29 04:00:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0680c6e7]) (logid:1e6e3d66) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:40,550 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e4a2787b]) (logid:05fc15c8) HA health check task is running... 2025-06-29 04:00:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34c6846b]) (logid:a91a9a7d) No inactive management server node found 2025-06-29 04:00:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34c6846b]) (logid:a91a9a7d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-f446ac9a]) (logid:bd1852fe) DbCollector is running... 2025-06-29 04:00:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fecbe51a]) (logid:8c71806d) Found 0 routers to update status. 2025-06-29 04:00:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fecbe51a]) (logid:8c71806d) Found 0 VPC's to update Redundant State. 2025-06-29 04:00:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fecbe51a]) (logid:8c71806d) Found 0 networks to update RvR status. 2025-06-29 04:00:43,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-846b423c]) (logid:99fbb082) No inactive management server node found 2025-06-29 04:00:43,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-846b423c]) (logid:99fbb082) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:44,042 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) ManagementServerCollector is running... 2025-06-29 04:00:44,044 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) DB properties were already loaded 2025-06-29 04:00:44,044 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:00:44,059 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Execution of process [4065834] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:00:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Exit value of process [4065834] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:00:44,060 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Process [4065834] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:00:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) usage install: null 2025-06-29 04:00:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Sessions active: 0 2025-06-29 04:00:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:00:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065837] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:00:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:00:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065842] for command [/bin/bash -c uname -r ]. 2025-06-29 04:00:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:00:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065843] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:00:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) system memory from /proc: 4887515136 2025-06-29 04:00:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:00:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065848] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:00:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) free memory from /proc: 300986368 2025-06-29 04:00:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:00:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065853] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:00:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) used memory from /proc: 1043792 2025-06-29 04:00:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:00:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065854] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:00:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:00:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065855] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:00:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:00:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065856] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:00:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:00:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065863] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:00:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:00:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Successfully executed process [4065864] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail buffersdirect.capacity, 318092 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail buffersdirect.count, 90 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail buffersdirect.used, 318092 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail buffersmapped.capacity, 0 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail buffersmapped.count, 0 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail buffersmapped.used, 0 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail gcPS-Scavenge.count, 106 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail gcPS-Scavenge.time, 514 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail jvmuptime, 1503835 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memoryheap.committed, 536870912 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memoryheap.init, 77594624 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memoryheap.usage, 0.22199136743961995 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorynon-heap.committed, 193921024 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorynon-heap.max, -1 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorynon-heap.usage, -1.907538E8 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorynon-heap.used, 190753800 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10548541222340709 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20339004232345786 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.Compressed-Class-Space.usage, 0.012369312345981598 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.Metaspace.usage, 0.9932109061170841 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.PS-Eden-Space.usage, 0.3381250367856612 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorytotal.committed, 730791936 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorytotal.init, 85262336 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorytotal.max, 1908932607 2025-06-29 04:00:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail memorytotal.used, 614520360 2025-06-29 04:00:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail threadsdeadlocks, [] 2025-06-29 04:00:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail threadsnew.count, 0 2025-06-29 04:00:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:00:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-0f88363c]) (logid:71b7a9ce) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":113835624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1503763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.75,"totalInit":85262336,"totalUsed":613789088,"totalCommitted":730791936,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":423766560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":300986368,"systemMemoryUsed":1043792,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.75,0.78,0.47],"systemCyclesUsage":[66404650,53827367,4589195568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:00:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 49, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":113835624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1503763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.75,"totalInit":85262336,"totalUsed":613789088,"totalCommitted":730791936,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":423766560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":300986368,"systemMemoryUsed":1043792,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.75,0.78,0.47],"systemCyclesUsage":[66404650,53827367,4589195568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:00:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:00:44,144 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-14:[ctx-a1d21b91]) (logid:6d5f6da9) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":113835624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1503763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.75,"totalInit":85262336,"totalUsed":613789088,"totalCommitted":730791936,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":423766560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":300986368,"systemMemoryUsed":1043792,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.75,0.78,0.47],"systemCyclesUsage":[66404650,53827367,4589195568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:00:44,144 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:00:44,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 49, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":113835624,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1503763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.75,"totalInit":85262336,"totalUsed":613789088,"totalCommitted":730791936,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":423766560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":300986368,"systemMemoryUsed":1043792,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.75,0.78,0.47],"systemCyclesUsage":[66404650,53827367,4589195568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:00:44,551 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-73111dc3]) (logid:eddfcf16) HA health check task is running... 2025-06-29 04:00:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc4e11d]) (logid:7cbd14a1) No inactive management server node found 2025-06-29 04:00:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc4e11d]) (logid:7cbd14a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:00:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:00:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c64eb2ef]) (logid:222aac04) No inactive management server node found 2025-06-29 04:00:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c64eb2ef]) (logid:222aac04) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:46,646 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:00:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:00:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:00:46,653 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:00:47,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:00:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:00:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:00:47,102 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:00:47,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11c951ce]) (logid:2d8abb52) No inactive management server node found 2025-06-29 04:00:47,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11c951ce]) (logid:2d8abb52) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-55899108]) (logid:7c9cdd33) ClusterDRS.poll is being called at 2025-06-29T02:01:00+0000 2025-06-29 04:00:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-55899108]) (logid:7c9cdd33) Removed 0 old drs migration plans 2025-06-29 04:00:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-29ad0fa4]) (logid:fbfbfd54) Resetting hosts suitable for reconnect 2025-06-29 04:00:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-29ad0fa4]) (logid:fbfbfd54) Completed resetting hosts suitable for reconnect 2025-06-29 04:00:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-29ad0fa4]) (logid:fbfbfd54) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:00:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-29ad0fa4]) (logid:fbfbfd54) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:00:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-29ad0fa4]) (logid:fbfbfd54) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:00:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-29ad0fa4]) (logid:fbfbfd54) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:00:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0e045d61]) (logid:f00e21b5) VM scheduler.poll is being called at 2025-06-29T02:01:00+0000 2025-06-29 04:00:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0e045d61]) (logid:f00e21b5) Got 0 scheduled jobs to be executed at 2025-06-29T02:01:00+0000 2025-06-29 04:00:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0e045d61]) (logid:f00e21b5) Cleaned up 0 VM scheduled job entries 2025-06-29 04:00:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d80def4a]) (logid:b47cbf3f) VM Operation Thread Running 2025-06-29 04:00:48,552 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-41bb6f3e]) (logid:f11063f1) HA health check task is running... 2025-06-29 04:00:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ecab86e]) (logid:6ee9e8a0) No inactive management server node found 2025-06-29 04:00:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ecab86e]) (logid:6ee9e8a0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:50,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8421: Seq 7-8421: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:00:50,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8421 2025-06-29 04:00:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa58cdbc]) (logid:34bbf537) No inactive management server node found 2025-06-29 04:00:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa58cdbc]) (logid:34bbf537) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b603415d]) (logid:5cfa8665) No inactive management server node found 2025-06-29 04:00:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b603415d]) (logid:5cfa8665) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:52,554 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4833edb3]) (logid:b0580787) HA health check task is running... 2025-06-29 04:00:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-654923a7]) (logid:2389179e) No inactive management server node found 2025-06-29 04:00:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-654923a7]) (logid:2389179e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:55,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12dfa08b]) (logid:b1f4f1ae) No inactive management server node found 2025-06-29 04:00:55,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12dfa08b]) (logid:b1f4f1ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:56,555 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ab9a61a4]) (logid:1f944ffc) HA health check task is running... 2025-06-29 04:00:56,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5422c8e]) (logid:615afb6a) No inactive management server node found 2025-06-29 04:00:56,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5422c8e]) (logid:615afb6a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c07db2db]) (logid:c795cc2c) Enabled non-edge zones available for scan: 1 2025-06-29 04:00:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c07db2db]) (logid:c795cc2c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:00:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c07db2db]) (logid:c795cc2c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:00:58,105 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-b0ecb3b3]) (logid:24e65ec7) AutoScaling Monitor is running... 2025-06-29 04:00:58,114 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-a170c84c]) (logid:ad573b87) Starting to remove all volume_stats rows older than [Sat Jun 28 16:00:58 CEST 2025]. 2025-06-29 04:00:58,115 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-a170c84c]) (logid:ad573b87) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:00:58 CEST 2025]. 2025-06-29 04:00:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ce536355]) (logid:7c9faa6d) Enabled non-edge zones available for scan: 1 2025-06-29 04:00:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ce536355]) (logid:7c9faa6d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:00:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61eb905f]) (logid:ffd35b41) No inactive management server node found 2025-06-29 04:00:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61eb905f]) (logid:ffd35b41) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:00:58,802 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-b339cf94]) (logid:aaac4080) Starting to remove all vm_stats rows older than [Sat Jun 28 16:00:58 CEST 2025]. 2025-06-29 04:00:58,837 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-b339cf94]) (logid:aaac4080) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:00:58 CEST 2025]. 2025-06-29 04:00:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f17abe90]) (logid:a0a60dbd) No inactive management server node found 2025-06-29 04:00:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f17abe90]) (logid:a0a60dbd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:00,137 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:01:00,139 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:01:00,141 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) Seq 2-4725683383995203693: Routed from 90520738888109 2025-06-29 04:01:00,185 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) Seq 1-4725683383995203693: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:01:00,186 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:01:00,187 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) Seq 2-3926857400090361974: Routed from 90520738888109 2025-06-29 04:01:00,231 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b8fa853f]) (logid:a90fa685) Seq 2-3926857400090361974: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:01:00,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8422: Seq 7-8422: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:00,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8422 2025-06-29 04:01:00,556 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-98188024]) (logid:f7190fa8) HA health check task is running... 2025-06-29 04:01:01,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa73099c]) (logid:9d0d756a) No inactive management server node found 2025-06-29 04:01:01,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa73099c]) (logid:9d0d756a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:02,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65589e09]) (logid:d0ead17c) No inactive management server node found 2025-06-29 04:01:02,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65589e09]) (logid:d0ead17c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:04,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59e7e242]) (logid:e6b806be) No inactive management server node found 2025-06-29 04:01:04,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59e7e242]) (logid:e6b806be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:04,557 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:01:04,558 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1754c5b6]) (logid:da9c5c13) HA health check task is running... 2025-06-29 04:01:04,561 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:01:04,562 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) Seq 2-4725683383995203694: Routed from 90520738888109 2025-06-29 04:01:04,616 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) Seq 1-4725683383995203694: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:01:04,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:01:04,652 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) Seq 2-3926857400090361975: Routed from 90520738888109 2025-06-29 04:01:04,743 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-b370e1aa]) (logid:36ed0fbe) Seq 2-3926857400090361975: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:01:05,521 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) StorageCollector is running... 2025-06-29 04:01:05,528 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:01:05,529 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:01:05,529 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:01:05,530 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-3938116399158788128: Routed from 90520738888109 2025-06-29 04:01:05,587 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 4-3938116399158788128: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:01:05,591 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:01:05,591 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:01:05,591 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:01:05,593 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-4725683383995203695: Routed from 90520738888109 2025-06-29 04:01:05,640 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 1-4725683383995203695: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:01:05,651 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:01:05,651 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:01:05,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:01:05,652 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-3926857400090361976: Routed from 90520738888109 2025-06-29 04:01:05,667 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-3926857400090361976: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:01:05,678 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:01:05,678 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:01:05,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:01:05,679 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-4725683383995203696: Routed from 90520738888109 2025-06-29 04:01:05,722 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 1-4725683383995203696: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:01:05,725 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:01:05,725 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:01:05,725 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:01:05,725 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-3926857400090361977: Routed from 90520738888109 2025-06-29 04:01:05,769 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2178a30f]) (logid:5d8c8e1a) Seq 2-3926857400090361977: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:01:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d578483a]) (logid:5ca14b92) No inactive management server node found 2025-06-29 04:01:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d578483a]) (logid:5ca14b92) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:07,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4998082a]) (logid:da0d1e56) No inactive management server node found 2025-06-29 04:01:07,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4998082a]) (logid:da0d1e56) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:08,559 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3909fdf4]) (logid:6fbb11b9) HA health check task is running... 2025-06-29 04:01:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0769bdf]) (logid:aaa3c08f) No inactive management server node found 2025-06-29 04:01:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0769bdf]) (logid:aaa3c08f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:10,271 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8423: Seq 7-8423: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:10,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8423 2025-06-29 04:01:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc9aa011]) (logid:02a6ecf2) No inactive management server node found 2025-06-29 04:01:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc9aa011]) (logid:02a6ecf2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e8eca59]) (logid:ca5e5cb5) No inactive management server node found 2025-06-29 04:01:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e8eca59]) (logid:ca5e5cb5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:12,560 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f095fc98]) (logid:cf46d444) HA health check task is running... 2025-06-29 04:01:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8e7e643c]) (logid:ee3a31f0) Found 0 routers to update status. 2025-06-29 04:01:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8e7e643c]) (logid:ee3a31f0) Found 0 VPC's to update Redundant State. 2025-06-29 04:01:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8e7e643c]) (logid:ee3a31f0) Found 0 networks to update RvR status. 2025-06-29 04:01:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fe6be48]) (logid:a37757f3) No inactive management server node found 2025-06-29 04:01:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fe6be48]) (logid:a37757f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cad6f9a]) (logid:a871a6b1) No inactive management server node found 2025-06-29 04:01:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cad6f9a]) (logid:a871a6b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:15,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8424: Seq 7-8424: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:15,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8424 2025-06-29 04:01:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77c034f]) (logid:07957892) No inactive management server node found 2025-06-29 04:01:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e77c034f]) (logid:07957892) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:16,562 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-f625e29f]) (logid:437da4d3) HA health check task is running... 2025-06-29 04:01:17,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1633b7a5]) (logid:d1fc44b0) No inactive management server node found 2025-06-29 04:01:17,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1633b7a5]) (logid:d1fc44b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8e11bb5]) (logid:cdc3a4fc) No inactive management server node found 2025-06-29 04:01:19,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8e11bb5]) (logid:cdc3a4fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:20,564 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-058e6b60]) (logid:7bbe267b) HA health check task is running... 2025-06-29 04:01:20,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5338982]) (logid:d960acba) No inactive management server node found 2025-06-29 04:01:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5338982]) (logid:d960acba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a39789e]) (logid:b627b035) No inactive management server node found 2025-06-29 04:01:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a39789e]) (logid:b627b035) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37ce09d7]) (logid:e1a3540f) No inactive management server node found 2025-06-29 04:01:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37ce09d7]) (logid:e1a3540f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:24,565 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1a17a58d]) (logid:10276411) HA health check task is running... 2025-06-29 04:01:25,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8425: Seq 7-8425: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:25,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8425 2025-06-29 04:01:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fd3722d]) (logid:fff145ec) No inactive management server node found 2025-06-29 04:01:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fd3722d]) (logid:fff145ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3258d6b]) (logid:4b8a0a57) No inactive management server node found 2025-06-29 04:01:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3258d6b]) (logid:4b8a0a57) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bb0a0d97]) (logid:6a555c92) Enabled non-edge zones available for scan: 1 2025-06-29 04:01:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bb0a0d97]) (logid:6a555c92) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:01:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bb0a0d97]) (logid:6a555c92) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:01:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fd5f48f0]) (logid:f455d3f9) Enabled non-edge zones available for scan: 1 2025-06-29 04:01:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-fd5f48f0]) (logid:f455d3f9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:01:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a88c6bf]) (logid:99c14e73) No inactive management server node found 2025-06-29 04:01:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a88c6bf]) (logid:99c14e73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:28,566 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-938420ee]) (logid:572cbb2e) HA health check task is running... 2025-06-29 04:01:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88248894]) (logid:3109cf58) No inactive management server node found 2025-06-29 04:01:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88248894]) (logid:3109cf58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:30,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8426: Seq 7-8426: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:30,283 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8426 2025-06-29 04:01:31,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-277e6bdd]) (logid:4aebcdca) No inactive management server node found 2025-06-29 04:01:31,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-277e6bdd]) (logid:4aebcdca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:32,568 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b95c7ac1]) (logid:716bf522) HA health check task is running... 2025-06-29 04:01:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bc5b0a9]) (logid:a55b03fe) No inactive management server node found 2025-06-29 04:01:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5bc5b0a9]) (logid:a55b03fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-911943f0]) (logid:dcc6864f) No inactive management server node found 2025-06-29 04:01:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-911943f0]) (logid:dcc6864f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9617e361]) (logid:c66c9548) No inactive management server node found 2025-06-29 04:01:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9617e361]) (logid:c66c9548) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:36,569 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e2c491b9]) (logid:cd02b0c8) HA health check task is running... 2025-06-29 04:01:37,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5c516d8]) (logid:5f3220ee) No inactive management server node found 2025-06-29 04:01:37,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5c516d8]) (logid:5f3220ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f32f80f8]) (logid:ad61e136) No inactive management server node found 2025-06-29 04:01:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f32f80f8]) (logid:ad61e136) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:40,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8427: Seq 7-8427: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:40,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8427 2025-06-29 04:01:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72346b12]) (logid:538374b0) No inactive management server node found 2025-06-29 04:01:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72346b12]) (logid:538374b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:40,570 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4cb6bda3]) (logid:aedf541c) HA health check task is running... 2025-06-29 04:01:41,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e786ae6]) (logid:2f2b048d) No inactive management server node found 2025-06-29 04:01:41,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e786ae6]) (logid:2f2b048d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-596fd71b]) (logid:76761ca4) DbCollector is running... 2025-06-29 04:01:43,094 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-7b872101]) (logid:c2b8d276) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:01:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-7b872101]) (logid:c2b8d276) Snapshot scheduler is being called at [2025-06-29T02:01:43+0000]. 2025-06-29 04:01:43,096 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-7b872101]) (logid:c2b8d276) There are [0] scheduled snapshots to be executed at [2025-06-29T02:01:43+0000]. 2025-06-29 04:01:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-3039bdad]) (logid:8e5601aa) Found 0 running routers. 2025-06-29 04:01:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cfcc6651]) (logid:c1dd057f) Found 0 routers to update status. 2025-06-29 04:01:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cfcc6651]) (logid:c1dd057f) Found 0 VPC's to update Redundant State. 2025-06-29 04:01:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-cfcc6651]) (logid:c1dd057f) Found 0 networks to update RvR status. 2025-06-29 04:01:43,118 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-84bd280e]) (logid:e0d934ac) External devices are not used. Skipping external device usage collection 2025-06-29 04:01:43,126 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-4b66d1af]) (logid:f600e757) Started resource counters recalculation periodic task. 2025-06-29 04:01:43,129 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-4b66d1af]) (logid:f600e757) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:01:43,304 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-53597362]) (logid:afcb131a) Backup backup.poll is being called at 2025-06-29T02:01:43+0000 2025-06-29 04:01:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b043878e]) (logid:2bb35bed) No inactive management server node found 2025-06-29 04:01:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b043878e]) (logid:2bb35bed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:43,498 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-4b66d1af]) (logid:f600e757) Finished resource counters recalculation periodic task. 2025-06-29 04:01:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) ManagementServerCollector is running... 2025-06-29 04:01:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) DB properties were already loaded 2025-06-29 04:01:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:01:44,058 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Execution of process [4065881] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:01:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Exit value of process [4065881] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:01:44,058 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Process [4065881] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:01:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) usage install: null 2025-06-29 04:01:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Sessions active: 0 2025-06-29 04:01:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:01:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065884] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:01:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:01:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065889] for command [/bin/bash -c uname -r ]. 2025-06-29 04:01:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:01:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065890] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:01:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) system memory from /proc: 4887515136 2025-06-29 04:01:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:01:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065895] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:01:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) free memory from /proc: 310218752 2025-06-29 04:01:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:01:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065900] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:01:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) used memory from /proc: 1043920 2025-06-29 04:01:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:01:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065901] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:01:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:01:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065902] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:01:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:01:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065903] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:01:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:01:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065910] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:01:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:01:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Successfully executed process [4065911] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail buffersdirect.capacity, 318101 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail buffersdirect.count, 90 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail buffersdirect.used, 318101 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail buffersmapped.capacity, 0 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail buffersmapped.count, 0 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail buffersmapped.used, 0 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail gcPS-Scavenge.count, 106 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail gcPS-Scavenge.time, 514 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail jvmuptime, 1563834 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memoryheap.committed, 536870912 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memoryheap.init, 77594624 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memoryheap.usage, 0.23949778744625017 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorynon-heap.committed, 193986560 2025-06-29 04:01:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorynon-heap.max, -1 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorynon-heap.usage, -1.90978536E8 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorynon-heap.used, 190978536 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10567702779258864 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20465328923251241 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.Compressed-Class-Space.usage, 0.012369312345981598 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.Metaspace.usage, 0.9935420420309546 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.PS-Eden-Space.usage, 0.38867291564699397 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorytotal.committed, 730857472 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorytotal.init, 85262336 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorytotal.max, 1908932607 2025-06-29 04:01:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail memorytotal.used, 648163672 2025-06-29 04:01:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail threadsdeadlocks, [] 2025-06-29 04:01:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail threadsnew.count, 0 2025-06-29 04:01:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:01:44,136 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-4ad8a06b]) (logid:188cfdd1) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:01:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":80456760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1563762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":647390328,"totalCommitted":730857472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":457185136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":310218752,"systemMemoryUsed":1043920,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.68,0.45],"systemCyclesUsage":[66405054,53827735,4589218536],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:01:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 51, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:01:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":80456760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1563762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":647390328,"totalCommitted":730857472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":457185136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":310218752,"systemMemoryUsed":1043920,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.68,0.45],"systemCyclesUsage":[66405054,53827735,4589218536],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:01:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:01:44,144 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-14:[ctx-7477edc4]) (logid:30f308ae) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:01:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":80456760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1563762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":647390328,"totalCommitted":730857472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":457185136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":310218752,"systemMemoryUsed":1043920,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.68,0.45],"systemCyclesUsage":[66405054,53827735,4589218536],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:01:44,145 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:01:44,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 51, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:01:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":80456760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1563762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":647390328,"totalCommitted":730857472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":457185136,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":310218752,"systemMemoryUsed":1043920,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.68,0.45],"systemCyclesUsage":[66405054,53827735,4589218536],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:01:44,572 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-26f5b764]) (logid:d3601187) HA health check task is running... 2025-06-29 04:01:44,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7caa260]) (logid:e3dbfa05) No inactive management server node found 2025-06-29 04:01:44,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7caa260]) (logid:e3dbfa05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:01:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:01:46,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed14ffc5]) (logid:9e76a56c) No inactive management server node found 2025-06-29 04:01:46,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed14ffc5]) (logid:9e76a56c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:01:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:01:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:01:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:01:47,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:01:47,102 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:01:47,104 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:01:47,107 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:01:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9d1809]) (logid:f3c2af91) No inactive management server node found 2025-06-29 04:01:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9d1809]) (logid:f3c2af91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:48,043 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-4:[ctx-8b979fb8]) (logid:4354e2c3) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:01:48,069 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-85c3d2f0]) (logid:840b25b2) ClusterDRS.poll is being called at 2025-06-29T02:02:00+0000 2025-06-29 04:01:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-85c3d2f0]) (logid:840b25b2) Removed 0 old drs migration plans 2025-06-29 04:01:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9ba38efe]) (logid:0e0440e4) VM scheduler.poll is being called at 2025-06-29T02:02:00+0000 2025-06-29 04:01:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9ba38efe]) (logid:0e0440e4) Got 0 scheduled jobs to be executed at 2025-06-29T02:02:00+0000 2025-06-29 04:01:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9ba38efe]) (logid:0e0440e4) Cleaned up 0 VM scheduled job entries 2025-06-29 04:01:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-7f9233c5]) (logid:35ac9c7b) VM Operation Thread Running 2025-06-29 04:01:48,573 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-14995631]) (logid:ad2db6de) HA health check task is running... 2025-06-29 04:01:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99a33434]) (logid:fa591012) No inactive management server node found 2025-06-29 04:01:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99a33434]) (logid:fa591012) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:50,273 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8429: Seq 7-8429: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:01:50,294 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8429 2025-06-29 04:01:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3c10e27]) (logid:5ba10b67) No inactive management server node found 2025-06-29 04:01:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3c10e27]) (logid:5ba10b67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-370d8ab5]) (logid:7333830c) No inactive management server node found 2025-06-29 04:01:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-370d8ab5]) (logid:7333830c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:52,575 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7a0ee385]) (logid:3f991a3c) HA health check task is running... 2025-06-29 04:01:53,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7572531]) (logid:c5251653) No inactive management server node found 2025-06-29 04:01:53,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7572531]) (logid:c5251653) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:55,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfa512b8]) (logid:2439309a) No inactive management server node found 2025-06-29 04:01:55,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfa512b8]) (logid:2439309a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:56,576 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-095185bf]) (logid:ec330324) HA health check task is running... 2025-06-29 04:01:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c53d30be]) (logid:083aab5b) No inactive management server node found 2025-06-29 04:01:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c53d30be]) (logid:083aab5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6adfb9a5]) (logid:f25030f2) Enabled non-edge zones available for scan: 1 2025-06-29 04:01:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6adfb9a5]) (logid:f25030f2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:01:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6adfb9a5]) (logid:f25030f2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:01:58,107 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-984703be]) (logid:54cb998e) AutoScaling Monitor is running... 2025-06-29 04:01:58,117 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-e9595421]) (logid:28df7070) Starting to remove all volume_stats rows older than [Sat Jun 28 16:01:58 CEST 2025]. 2025-06-29 04:01:58,118 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-e9595421]) (logid:28df7070) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:01:58 CEST 2025]. 2025-06-29 04:01:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1e6b1f8e]) (logid:146a9cc4) Enabled non-edge zones available for scan: 1 2025-06-29 04:01:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1e6b1f8e]) (logid:146a9cc4) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:01:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1996a5bc]) (logid:02e31d36) No inactive management server node found 2025-06-29 04:01:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1996a5bc]) (logid:02e31d36) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:01:58,839 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-7bc3c7be]) (logid:138926dc) Starting to remove all vm_stats rows older than [Sat Jun 28 16:01:58 CEST 2025]. 2025-06-29 04:01:58,872 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-7bc3c7be]) (logid:138926dc) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:01:58 CEST 2025]. 2025-06-29 04:01:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cd85d8f]) (logid:12ac6d3f) No inactive management server node found 2025-06-29 04:01:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cd85d8f]) (logid:12ac6d3f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:00,235 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:02:00,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:02:00,238 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) Seq 2-4725683383995203697: Routed from 90520738888109 2025-06-29 04:02:00,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8430: Seq 7-8430: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:00,282 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) Seq 1-4725683383995203697: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:02:00,283 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:02:00,284 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) Seq 2-3926857400090361978: Routed from 90520738888109 2025-06-29 04:02:00,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8430 2025-06-29 04:02:00,329 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-202c5fb6]) (logid:d4ab41f7) Seq 2-3926857400090361978: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:02:00,577 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-9a44b5ee]) (logid:4cc922fe) HA health check task is running... 2025-06-29 04:02:01,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a2e36d]) (logid:a1f7ade9) No inactive management server node found 2025-06-29 04:02:01,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a2e36d]) (logid:a1f7ade9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:02,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-445fad26]) (logid:5088ee11) No inactive management server node found 2025-06-29 04:02:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-445fad26]) (logid:5088ee11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:04,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38955b3c]) (logid:e7291de7) No inactive management server node found 2025-06-29 04:02:04,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38955b3c]) (logid:e7291de7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:02:04,579 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d1c18ec8]) (logid:8ea69563) HA health check task is running... 2025-06-29 04:02:04,797 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:02:04,800 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:02:04,801 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) Seq 2-4725683383995203698: Routed from 90520738888109 2025-06-29 04:02:04,855 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) Seq 1-4725683383995203698: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:02:04,902 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:02:04,904 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) Seq 2-3926857400090361979: Routed from 90520738888109 2025-06-29 04:02:05,011 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-44d27b00]) (logid:e8e4b52d) Seq 2-3926857400090361979: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:02:05,770 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) StorageCollector is running... 2025-06-29 04:02:05,775 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:02:05,776 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:02:05,776 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:02:05,777 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-3938116399158788129: Routed from 90520738888109 2025-06-29 04:02:05,853 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 4-3938116399158788129: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:02:05,857 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:02:05,857 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:02:05,857 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:02:05,858 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-3926857400090361980: Routed from 90520738888109 2025-06-29 04:02:05,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4940e8e9]) (logid:c341fd4f) No inactive management server node found 2025-06-29 04:02:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4940e8e9]) (logid:c341fd4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:05,873 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-3926857400090361980: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:02:05,883 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:02:05,883 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:02:05,883 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:02:05,884 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-4725683383995203699: Routed from 90520738888109 2025-06-29 04:02:05,932 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 1-4725683383995203699: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:02:05,942 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:02:05,942 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:02:05,942 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:02:05,943 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-4725683383995203700: Routed from 90520738888109 2025-06-29 04:02:06,027 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 1-4725683383995203700: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:02:06,029 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:02:06,029 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:02:06,029 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:02:06,030 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-3926857400090361981: Routed from 90520738888109 2025-06-29 04:02:06,077 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b8e83365]) (logid:9aaf0770) Seq 2-3926857400090361981: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:02:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5239b5c9]) (logid:63ca21a6) No inactive management server node found 2025-06-29 04:02:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5239b5c9]) (logid:63ca21a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:08,580 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2a1eaf5b]) (logid:bee050da) HA health check task is running... 2025-06-29 04:02:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9886993b]) (logid:23355581) No inactive management server node found 2025-06-29 04:02:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9886993b]) (logid:23355581) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:10,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8431: Seq 7-8431: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:10,284 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8431 2025-06-29 04:02:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-075c61b8]) (logid:5a7cfb8b) No inactive management server node found 2025-06-29 04:02:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-075c61b8]) (logid:5a7cfb8b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:11,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c099a0f1]) (logid:0158e3a2) No inactive management server node found 2025-06-29 04:02:11,874 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c099a0f1]) (logid:0158e3a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:12,582 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-778d6170]) (logid:eea5367c) HA health check task is running... 2025-06-29 04:02:13,003 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-d285f5fd]) (logid:620a05b4) Checking if any host reservation can be released ... 2025-06-29 04:02:13,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-d285f5fd]) (logid:620a05b4) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-d285f5fd]) (logid:620a05b4) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-d285f5fd]) (logid:620a05b4) Done running HostReservationReleaseChecker ... 2025-06-29 04:02:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9fe8530]) (logid:4db02a6d) Found 0 routers to update status. 2025-06-29 04:02:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9fe8530]) (logid:4db02a6d) Found 0 VPC's to update Redundant State. 2025-06-29 04:02:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c9fe8530]) (logid:4db02a6d) Found 0 networks to update RvR status. 2025-06-29 04:02:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Running Capacity Checker ... 2025-06-29 04:02:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) recalculating system capacity 2025-06-29 04:02:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Executing cpu/ram capacity update 2025-06-29 04:02:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:13,181 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:13,187 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:02:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:02:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:02:13,200 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:13,201 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:13,208 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:13,210 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:02:13,210 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:02:13,211 DEBUG [c.c.c.CapacityManagerImpl] (pool-22-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:02:13,221 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Done executing cpu/ram capacity update 2025-06-29 04:02:13,221 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Executing storage capacity update 2025-06-29 04:02:13,231 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:02:13,231 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:02:13,231 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:02:13,234 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:02:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:02:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:02:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:02:13,244 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:02:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:02:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:02:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:02:13,255 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:02:13,261 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:02:13,261 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:02:13,261 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:02:13,265 DEBUG [c.c.s.StorageManagerImpl] (pool-23-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:02:13,265 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Done executing storage capacity update 2025-06-29 04:02:13,265 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Executing capacity updates for public ip and Vlans 2025-06-29 04:02:13,283 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Done capacity updates for public ip and Vlans 2025-06-29 04:02:13,283 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Executing capacity updates for private ip 2025-06-29 04:02:13,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Done executing capacity updates for private ip 2025-06-29 04:02:13,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Done recalculating system capacity 2025-06-29 04:02:13,329 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0595aaae]) (logid:af6a5bf1) Done running Capacity Checker ... 2025-06-29 04:02:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4230fef2]) (logid:f8d4979a) No inactive management server node found 2025-06-29 04:02:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4230fef2]) (logid:f8d4979a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e5ff120]) (logid:d0d04e67) No inactive management server node found 2025-06-29 04:02:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e5ff120]) (logid:d0d04e67) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:15,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8432: Seq 7-8432: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:15,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8432 2025-06-29 04:02:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eefeaeca]) (logid:4188b296) No inactive management server node found 2025-06-29 04:02:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eefeaeca]) (logid:4188b296) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:16,583 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4105cbec]) (logid:ce359762) HA health check task is running... 2025-06-29 04:02:17,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c343fb8e]) (logid:fc78c097) No inactive management server node found 2025-06-29 04:02:17,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c343fb8e]) (logid:fc78c097) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d59d7233]) (logid:8282c35a) Resetting hosts suitable for reconnect 2025-06-29 04:02:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d59d7233]) (logid:8282c35a) Completed resetting hosts suitable for reconnect 2025-06-29 04:02:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d59d7233]) (logid:8282c35a) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:02:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d59d7233]) (logid:8282c35a) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:02:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d59d7233]) (logid:8282c35a) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:02:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d59d7233]) (logid:8282c35a) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:02:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4edb09a]) (logid:85baf456) No inactive management server node found 2025-06-29 04:02:19,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4edb09a]) (logid:85baf456) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:20,584 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9bfc97e8]) (logid:72c751f0) HA health check task is running... 2025-06-29 04:02:20,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd4f43f7]) (logid:95f20e7a) No inactive management server node found 2025-06-29 04:02:20,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd4f43f7]) (logid:95f20e7a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3789d8b1]) (logid:b5f3b105) No inactive management server node found 2025-06-29 04:02:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3789d8b1]) (logid:b5f3b105) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:23,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-585e3ac3]) (logid:457cc135) No inactive management server node found 2025-06-29 04:02:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-585e3ac3]) (logid:457cc135) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:24,586 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2a2084fc]) (logid:919d4b3e) HA health check task is running... 2025-06-29 04:02:25,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8433: Seq 7-8433: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:25,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8433 2025-06-29 04:02:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-554227ea]) (logid:6f5c1540) No inactive management server node found 2025-06-29 04:02:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-554227ea]) (logid:6f5c1540) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:26,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3ff275a]) (logid:d8ad96cc) No inactive management server node found 2025-06-29 04:02:26,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3ff275a]) (logid:d8ad96cc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e74caf82]) (logid:584743fc) Enabled non-edge zones available for scan: 1 2025-06-29 04:02:28,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e74caf82]) (logid:584743fc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:02:28,100 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e74caf82]) (logid:584743fc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:02:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c2b322f5]) (logid:56d17919) Enabled non-edge zones available for scan: 1 2025-06-29 04:02:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c2b322f5]) (logid:56d17919) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:02:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f98c6f0]) (logid:6c89eeea) No inactive management server node found 2025-06-29 04:02:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f98c6f0]) (logid:6c89eeea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:28,588 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6100405e]) (logid:f7c0d29c) HA health check task is running... 2025-06-29 04:02:29,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-786b88e2]) (logid:2af1402d) No inactive management server node found 2025-06-29 04:02:29,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-786b88e2]) (logid:2af1402d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18fe6289]) (logid:7ed0f8d5) No inactive management server node found 2025-06-29 04:02:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18fe6289]) (logid:7ed0f8d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:32,589 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ddc5f565]) (logid:eb2647bf) HA health check task is running... 2025-06-29 04:02:32,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec3d1c87]) (logid:126c4079) No inactive management server node found 2025-06-29 04:02:32,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec3d1c87]) (logid:126c4079) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:34,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2059fc66]) (logid:b9592725) No inactive management server node found 2025-06-29 04:02:34,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2059fc66]) (logid:b9592725) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:35,275 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8434: Seq 7-8434: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:35,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8434 2025-06-29 04:02:35,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dced397]) (logid:bd959c09) No inactive management server node found 2025-06-29 04:02:35,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dced397]) (logid:bd959c09) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:36,590 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-968e64dc]) (logid:1cb99039) HA health check task is running... 2025-06-29 04:02:37,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e6316fd]) (logid:a5de9ecd) No inactive management server node found 2025-06-29 04:02:37,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e6316fd]) (logid:a5de9ecd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:38,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b36a71e7]) (logid:edde7f5b) No inactive management server node found 2025-06-29 04:02:38,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b36a71e7]) (logid:edde7f5b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:40,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7604796]) (logid:adc39799) No inactive management server node found 2025-06-29 04:02:40,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7604796]) (logid:adc39799) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:40,591 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e8299035]) (logid:388a0821) HA health check task is running... 2025-06-29 04:02:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bd427ea]) (logid:64cfdb17) No inactive management server node found 2025-06-29 04:02:41,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bd427ea]) (logid:64cfdb17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-4c843fca]) (logid:fa5f4565) DbCollector is running... 2025-06-29 04:02:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-281d4a40]) (logid:09dd9414) Found 0 routers to update status. 2025-06-29 04:02:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-281d4a40]) (logid:09dd9414) Found 0 VPC's to update Redundant State. 2025-06-29 04:02:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-281d4a40]) (logid:09dd9414) Found 0 networks to update RvR status. 2025-06-29 04:02:43,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27bb3dac]) (logid:9d8b8bd5) No inactive management server node found 2025-06-29 04:02:43,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27bb3dac]) (logid:9d8b8bd5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) ManagementServerCollector is running... 2025-06-29 04:02:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) DB properties were already loaded 2025-06-29 04:02:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:02:44,059 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Execution of process [4065927] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:02:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Exit value of process [4065927] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:02:44,060 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Process [4065927] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:02:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) usage install: null 2025-06-29 04:02:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Sessions active: 0 2025-06-29 04:02:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:02:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065930] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:02:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:02:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065935] for command [/bin/bash -c uname -r ]. 2025-06-29 04:02:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:02:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065936] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:02:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) system memory from /proc: 4887515136 2025-06-29 04:02:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:02:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065941] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:02:44,084 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) free memory from /proc: 319238144 2025-06-29 04:02:44,085 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:02:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065946] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:02:44,096 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) used memory from /proc: 1044416 2025-06-29 04:02:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:02:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065947] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:02:44,102 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:02:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065948] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:02:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:02:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065949] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:02:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:02:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065956] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:02:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:02:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Successfully executed process [4065957] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail buffersdirect.capacity, 318101 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail buffersdirect.count, 90 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail buffersdirect.used, 318101 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail buffersmapped.capacity, 0 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail buffersmapped.count, 0 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail buffersmapped.used, 0 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail gcPS-Scavenge.count, 106 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail gcPS-Scavenge.time, 514 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail jvmuptime, 1623835 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memoryheap.committed, 536870912 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memoryheap.init, 77594624 2025-06-29 04:02:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memoryheap.usage, 0.25502377504570345 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorynon-heap.committed, 194052096 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorynon-heap.max, -1 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorynon-heap.usage, -1.9117708E8 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorynon-heap.used, 191177080 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.106037348373767 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.20564680741160396 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.Compressed-Class-Space.usage, 0.012370340526103973 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.Metaspace.usage, 0.9932964542985356 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.PS-Eden-Space.usage, 0.43350251187983246 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorytotal.committed, 730923008 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorytotal.init, 85262336 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorytotal.max, 1908932607 2025-06-29 04:02:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail memorytotal.used, 678000280 2025-06-29 04:02:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail threadsdeadlocks, [] 2025-06-29 04:02:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail threadsnew.count, 0 2025-06-29 04:02:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:02:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-a0676595]) (logid:ac6fbfa1) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:02:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":50609608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1623764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":677436208,"totalCommitted":730923008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":486823200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":319238144,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.58,0.43],"systemCyclesUsage":[66405408,53828041,4589241626],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:02:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 53, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:02:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":50609608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1623764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":677436208,"totalCommitted":730923008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":486823200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":319238144,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.58,0.43],"systemCyclesUsage":[66405408,53828041,4589241626],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:02:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:02:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-14:[ctx-27b6e721]) (logid:3227449c) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:02:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":50609608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1623764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":677436208,"totalCommitted":730923008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":486823200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":319238144,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.58,0.43],"systemCyclesUsage":[66405408,53828041,4589241626],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:02:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:02:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 53, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:02:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":50609608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1623764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":677436208,"totalCommitted":730923008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":486823200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":108,"systemMemoryTotal":4887515136,"systemMemoryFree":319238144,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.58,0.43],"systemCyclesUsage":[66405408,53828041,4589241626],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:02:44,593 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-62f6046b]) (logid:3c0de0dc) HA health check task is running... 2025-06-29 04:02:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0f6977c]) (logid:3c19d1bc) No inactive management server node found 2025-06-29 04:02:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0f6977c]) (logid:3c19d1bc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:45,276 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8435: Seq 7-8435: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:45,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8435 2025-06-29 04:02:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:02:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:02:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cd0461b]) (logid:90a9802b) No inactive management server node found 2025-06-29 04:02:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cd0461b]) (logid:90a9802b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:02:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:02:47,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:47,087 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:02:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:02:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4282f66d]) (logid:14203068) No inactive management server node found 2025-06-29 04:02:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4282f66d]) (logid:14203068) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-34b21c76]) (logid:e7961f2e) ClusterDRS.poll is being called at 2025-06-29T02:03:00+0000 2025-06-29 04:02:48,076 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-34b21c76]) (logid:e7961f2e) Removed 0 old drs migration plans 2025-06-29 04:02:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3d9026e7]) (logid:877eec4f) VM scheduler.poll is being called at 2025-06-29T02:03:00+0000 2025-06-29 04:02:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3d9026e7]) (logid:877eec4f) Got 0 scheduled jobs to be executed at 2025-06-29T02:03:00+0000 2025-06-29 04:02:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3d9026e7]) (logid:877eec4f) Cleaned up 0 VM scheduled job entries 2025-06-29 04:02:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-50d299c6]) (logid:3e0ee30c) VM Operation Thread Running 2025-06-29 04:02:48,594 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ba004796]) (logid:95d77635) HA health check task is running... 2025-06-29 04:02:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5472bb1]) (logid:a8b1b6a6) No inactive management server node found 2025-06-29 04:02:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5472bb1]) (logid:a8b1b6a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bd9732d]) (logid:8723d7c3) No inactive management server node found 2025-06-29 04:02:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bd9732d]) (logid:8723d7c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd3cc681]) (logid:4b896c22) No inactive management server node found 2025-06-29 04:02:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd3cc681]) (logid:4b896c22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:52,595 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5e69c69f]) (logid:0715b1ec) HA health check task is running... 2025-06-29 04:02:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf67748f]) (logid:4f186e53) No inactive management server node found 2025-06-29 04:02:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf67748f]) (logid:4f186e53) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:55,276 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8437: Seq 7-8437: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:02:55,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8437 2025-06-29 04:02:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d72dfb85]) (logid:07a83aff) No inactive management server node found 2025-06-29 04:02:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d72dfb85]) (logid:07a83aff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:56,596 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-02227cbb]) (logid:044c68fe) HA health check task is running... 2025-06-29 04:02:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b569dab7]) (logid:782d101f) No inactive management server node found 2025-06-29 04:02:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b569dab7]) (logid:782d101f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-527e9931]) (logid:367d68c8) Enabled non-edge zones available for scan: 1 2025-06-29 04:02:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-527e9931]) (logid:367d68c8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:02:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-527e9931]) (logid:367d68c8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:02:58,109 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-d6ae9740]) (logid:d706073c) AutoScaling Monitor is running... 2025-06-29 04:02:58,120 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-d852b077]) (logid:f923f45f) Starting to remove all volume_stats rows older than [Sat Jun 28 16:02:58 CEST 2025]. 2025-06-29 04:02:58,120 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-d852b077]) (logid:f923f45f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:02:58 CEST 2025]. 2025-06-29 04:02:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-732ad484]) (logid:ac0dfc6f) Enabled non-edge zones available for scan: 1 2025-06-29 04:02:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-732ad484]) (logid:ac0dfc6f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:02:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46053544]) (logid:54372186) No inactive management server node found 2025-06-29 04:02:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46053544]) (logid:54372186) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:02:58,874 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a40edcfc]) (logid:537db62f) Starting to remove all vm_stats rows older than [Sat Jun 28 16:02:58 CEST 2025]. 2025-06-29 04:02:58,909 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a40edcfc]) (logid:537db62f) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:02:58 CEST 2025]. 2025-06-29 04:02:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bff947d]) (logid:be480bc6) No inactive management server node found 2025-06-29 04:02:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bff947d]) (logid:be480bc6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:00,333 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:03:00,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:03:00,337 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) Seq 2-4725683383995203701: Routed from 90520738888109 2025-06-29 04:03:00,381 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) Seq 1-4725683383995203701: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:03:00,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:03:00,383 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) Seq 2-3926857400090361982: Routed from 90520738888109 2025-06-29 04:03:00,427 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-543f9e97]) (logid:e602beb5) Seq 2-3926857400090361982: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:03:00,598 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9b99a68a]) (logid:00521b9e) HA health check task is running... 2025-06-29 04:03:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9988cdcd]) (logid:7a159c6d) No inactive management server node found 2025-06-29 04:03:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9988cdcd]) (logid:7a159c6d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:02,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffe22ed1]) (logid:bc3828ac) No inactive management server node found 2025-06-29 04:03:02,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffe22ed1]) (logid:bc3828ac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61977d1a]) (logid:efac1672) No inactive management server node found 2025-06-29 04:03:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61977d1a]) (logid:efac1672) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:04,599 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-6d069d57]) (logid:e38658ed) HA health check task is running... 2025-06-29 04:03:05,069 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:03:05,072 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:03:05,072 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) Seq 2-4725683383995203702: Routed from 90520738888109 2025-06-29 04:03:05,126 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) Seq 1-4725683383995203702: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:03:05,160 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:03:05,161 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) Seq 2-3926857400090361983: Routed from 90520738888109 2025-06-29 04:03:05,259 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-2d2b7dd9]) (logid:a7e25aad) Seq 2-3926857400090361983: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:03:05,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8438: Seq 7-8438: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:05,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8438 2025-06-29 04:03:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eda97915]) (logid:76fa8897) No inactive management server node found 2025-06-29 04:03:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eda97915]) (logid:76fa8897) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:06,077 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) StorageCollector is running... 2025-06-29 04:03:06,082 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:03:06,082 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:03:06,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:03:06,083 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 2-3938116399158788130: Routed from 90520738888109 2025-06-29 04:03:06,142 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 4-3938116399158788130: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:03:06,145 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:03:06,145 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:03:06,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:03:06,146 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 2-4725683383995203703: Routed from 90520738888109 2025-06-29 04:03:06,194 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 1-4725683383995203703: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:03:06,206 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:03:06,207 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:03:06,207 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:03:06,208 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 2-4725683383995203704: Routed from 90520738888109 2025-06-29 04:03:06,297 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 1-4725683383995203704: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:03:06,308 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:03:06,308 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:03:06,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:03:06,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 2-4725683383995203705: Routed from 90520738888109 2025-06-29 04:03:06,395 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 1-4725683383995203705: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:03:06,397 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:03:06,397 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:03:06,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:03:06,399 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 2-3926857400090361984: Routed from 90520738888109 2025-06-29 04:03:06,402 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3cb8c9e3]) (logid:d231173e) Seq 2-3926857400090361984: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:03:07,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16f1e727]) (logid:27c1a346) No inactive management server node found 2025-06-29 04:03:07,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16f1e727]) (logid:27c1a346) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:08,600 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f2454be9]) (logid:967fdb20) HA health check task is running... 2025-06-29 04:03:08,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5879a737]) (logid:a8710dba) No inactive management server node found 2025-06-29 04:03:08,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5879a737]) (logid:a8710dba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-534012ba]) (logid:dc88ce42) No inactive management server node found 2025-06-29 04:03:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-534012ba]) (logid:dc88ce42) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e191232b]) (logid:014d97b4) No inactive management server node found 2025-06-29 04:03:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e191232b]) (logid:014d97b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:12,602 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-85d96e80]) (logid:6274106a) HA health check task is running... 2025-06-29 04:03:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c5413e9]) (logid:f3790916) Found 0 routers to update status. 2025-06-29 04:03:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c5413e9]) (logid:f3790916) Found 0 VPC's to update Redundant State. 2025-06-29 04:03:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c5413e9]) (logid:f3790916) Found 0 networks to update RvR status. 2025-06-29 04:03:13,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-248bbe5e]) (logid:27d2335c) No inactive management server node found 2025-06-29 04:03:13,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-248bbe5e]) (logid:27d2335c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:14,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd4b5df3]) (logid:94aa4b63) No inactive management server node found 2025-06-29 04:03:14,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd4b5df3]) (logid:94aa4b63) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:15,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8439: Seq 7-8439: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:15,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8439 2025-06-29 04:03:16,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b88c4bbb]) (logid:0dc6ba97) No inactive management server node found 2025-06-29 04:03:16,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b88c4bbb]) (logid:0dc6ba97) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:16,603 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-87dd2496]) (logid:7c923405) HA health check task is running... 2025-06-29 04:03:17,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b8c6873]) (logid:52fa66e1) No inactive management server node found 2025-06-29 04:03:17,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b8c6873]) (logid:52fa66e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:19,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-148becb0]) (logid:9eed74ca) No inactive management server node found 2025-06-29 04:03:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-148becb0]) (logid:9eed74ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:20,605 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ada7149c]) (logid:b9ecf513) HA health check task is running... 2025-06-29 04:03:20,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2627d381]) (logid:8accbe8e) No inactive management server node found 2025-06-29 04:03:20,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2627d381]) (logid:8accbe8e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:22,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69e886a3]) (logid:13358f6a) No inactive management server node found 2025-06-29 04:03:22,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69e886a3]) (logid:13358f6a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:23,875 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff77774d]) (logid:b67dd59e) No inactive management server node found 2025-06-29 04:03:23,875 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff77774d]) (logid:b67dd59e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:24,606 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e69d8c96]) (logid:93f01a88) HA health check task is running... 2025-06-29 04:03:25,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8440: Seq 7-8440: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:25,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8440 2025-06-29 04:03:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-877b8a5a]) (logid:72d7b294) No inactive management server node found 2025-06-29 04:03:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-877b8a5a]) (logid:72d7b294) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:26,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-320a3c79]) (logid:e92e7033) No inactive management server node found 2025-06-29 04:03:26,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-320a3c79]) (logid:e92e7033) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9ee4d03a]) (logid:c9911ebc) Enabled non-edge zones available for scan: 1 2025-06-29 04:03:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9ee4d03a]) (logid:c9911ebc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:03:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9ee4d03a]) (logid:c9911ebc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:03:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3bc20947]) (logid:306dada3) Enabled non-edge zones available for scan: 1 2025-06-29 04:03:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3bc20947]) (logid:306dada3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:03:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e81e808]) (logid:41a9070c) No inactive management server node found 2025-06-29 04:03:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e81e808]) (logid:41a9070c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:28,608 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-984fe3f7]) (logid:05b3f940) HA health check task is running... 2025-06-29 04:03:29,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff0c3575]) (logid:a63e192d) No inactive management server node found 2025-06-29 04:03:29,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff0c3575]) (logid:a63e192d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:30,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8441: Seq 7-8441: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:30,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8441 2025-06-29 04:03:31,377 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec56e869]) (logid:5cb30b47) No inactive management server node found 2025-06-29 04:03:31,377 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec56e869]) (logid:5cb30b47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:32,609 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2630bb3c]) (logid:5f5a433c) HA health check task is running... 2025-06-29 04:03:32,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d808862]) (logid:d7ee4b11) No inactive management server node found 2025-06-29 04:03:32,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d808862]) (logid:d7ee4b11) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:34,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d44232cf]) (logid:91a95c60) No inactive management server node found 2025-06-29 04:03:34,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d44232cf]) (logid:91a95c60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:35,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6237c21]) (logid:8310e379) No inactive management server node found 2025-06-29 04:03:35,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6237c21]) (logid:8310e379) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:36,611 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-04a5de72]) (logid:db9ab078) HA health check task is running... 2025-06-29 04:03:37,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88ff12a3]) (logid:56e92a73) No inactive management server node found 2025-06-29 04:03:37,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88ff12a3]) (logid:56e92a73) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:03:38,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94a42ce0]) (logid:6ba30e07) No inactive management server node found 2025-06-29 04:03:38,869 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94a42ce0]) (logid:6ba30e07) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:40,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8442: Seq 7-8442: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:40,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8442 2025-06-29 04:03:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c832d9a1]) (logid:1776c514) No inactive management server node found 2025-06-29 04:03:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c832d9a1]) (logid:1776c514) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:40,612 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-94fbff3c]) (logid:334eb7f7) HA health check task is running... 2025-06-29 04:03:41,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1018a029]) (logid:73022c49) No inactive management server node found 2025-06-29 04:03:41,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1018a029]) (logid:73022c49) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-72fc8104]) (logid:eb5dda7b) DbCollector is running... 2025-06-29 04:03:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-74b308ac]) (logid:10bdc607) Found 0 routers to update status. 2025-06-29 04:03:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-74b308ac]) (logid:10bdc607) Found 0 VPC's to update Redundant State. 2025-06-29 04:03:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-74b308ac]) (logid:10bdc607) Found 0 networks to update RvR status. 2025-06-29 04:03:43,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab58dc6d]) (logid:8493bf97) No inactive management server node found 2025-06-29 04:03:43,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab58dc6d]) (logid:8493bf97) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) ManagementServerCollector is running... 2025-06-29 04:03:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) DB properties were already loaded 2025-06-29 04:03:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:03:44,056 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Execution of process [4065963] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:03:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Exit value of process [4065963] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:03:44,056 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Process [4065963] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:03:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) usage install: null 2025-06-29 04:03:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Sessions active: 0 2025-06-29 04:03:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:03:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065966] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:03:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:03:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065971] for command [/bin/bash -c uname -r ]. 2025-06-29 04:03:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:03:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065972] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:03:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) system memory from /proc: 4887515136 2025-06-29 04:03:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:03:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065977] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:03:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) free memory from /proc: 329764864 2025-06-29 04:03:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:03:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065982] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:03:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) used memory from /proc: 1044416 2025-06-29 04:03:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:03:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065983] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:03:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:03:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065984] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:03:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:03:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065985] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:03:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:03:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065992] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:03:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:03:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Successfully executed process [4065993] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail buffersdirect.capacity, 318202 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail buffersdirect.count, 91 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail buffersdirect.used, 318202 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail buffersmapped.capacity, 0 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail buffersmapped.count, 0 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail buffersmapped.used, 0 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail gcPS-Scavenge.count, 106 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail gcPS-Scavenge.time, 514 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail jvmuptime, 1683828 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memoryheap.committed, 536870912 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memoryheap.init, 77594624 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memoryheap.usage, 0.266970535190313 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorynon-heap.committed, 194117632 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:03:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorynon-heap.max, -1 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorynon-heap.usage, -1.9126012E8 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorynon-heap.used, 191260120 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10626332977872567 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2060529626420502 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.Compressed-Class-Space.usage, 0.012370340526103973 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.Metaspace.usage, 0.9933349724698867 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.PS-Eden-Space.usage, 0.46799747911736095 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.PS-Old-Gen.usage, 0.11203119144558427 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorypools.PS-Survivor-Space.usage, 0.9991804424085116 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorytotal.committed, 730988544 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorytotal.init, 85262336 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorytotal.max, 1908932607 2025-06-29 04:03:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail memorytotal.used, 700893232 2025-06-29 04:03:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail threadsdeadlocks, [] 2025-06-29 04:03:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail threadsnew.count, 0 2025-06-29 04:03:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:03:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-24797a6f]) (logid:342ab612) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:03:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":27973368,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1683760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.13,"totalInit":85262336,"totalUsed":700156680,"totalCommitted":730988544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":509628760,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":329764864,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.13,0.49,0.41],"systemCyclesUsage":[66405758,53828380,4589264687],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:03:44,130 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 55, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:03:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":27973368,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1683760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.13,"totalInit":85262336,"totalUsed":700156680,"totalCommitted":730988544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":509628760,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":329764864,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.13,0.49,0.41],"systemCyclesUsage":[66405758,53828380,4589264687],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:03:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:03:44,136 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-14:[ctx-d695d15c]) (logid:fa9c48fa) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:03:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":27973368,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1683760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.13,"totalInit":85262336,"totalUsed":700156680,"totalCommitted":730988544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":509628760,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":329764864,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.13,0.49,0.41],"systemCyclesUsage":[66405758,53828380,4589264687],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:03:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:03:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 55, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:03:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":536870912,"freeJvmMemoryBytes":27973368,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1683760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.13,"totalInit":85262336,"totalUsed":700156680,"totalCommitted":730988544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":509628760,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":329764864,"systemMemoryUsed":1044416,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.13,0.49,0.41],"systemCyclesUsage":[66405758,53828380,4589264687],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:03:44,613 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0108ed66]) (logid:43966463) HA health check task is running... 2025-06-29 04:03:44,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd5363b7]) (logid:a339d686) No inactive management server node found 2025-06-29 04:03:44,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd5363b7]) (logid:a339d686) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:03:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:03:46,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebe3a35d]) (logid:b96609be) No inactive management server node found 2025-06-29 04:03:46,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebe3a35d]) (logid:b96609be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:03:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:03:46,683 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:03:46,684 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:03:46,685 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:03:46,688 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:03:47,100 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:03:47,101 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:03:47,103 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:03:47,105 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:03:47,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-551637bd]) (logid:61a35d69) No inactive management server node found 2025-06-29 04:03:47,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-551637bd]) (logid:61a35d69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:48,068 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-dd16f206]) (logid:02133e80) ClusterDRS.poll is being called at 2025-06-29T02:04:00+0000 2025-06-29 04:03:48,075 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-dd16f206]) (logid:02133e80) Removed 0 old drs migration plans 2025-06-29 04:03:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-55668fa1]) (logid:1bbbe0a8) Resetting hosts suitable for reconnect 2025-06-29 04:03:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-55668fa1]) (logid:1bbbe0a8) Completed resetting hosts suitable for reconnect 2025-06-29 04:03:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-55668fa1]) (logid:1bbbe0a8) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:03:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-55668fa1]) (logid:1bbbe0a8) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:03:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-55668fa1]) (logid:1bbbe0a8) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:03:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-55668fa1]) (logid:1bbbe0a8) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:03:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6dc0077f]) (logid:4152e55e) VM scheduler.poll is being called at 2025-06-29T02:04:00+0000 2025-06-29 04:03:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6dc0077f]) (logid:4152e55e) Got 0 scheduled jobs to be executed at 2025-06-29T02:04:00+0000 2025-06-29 04:03:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6dc0077f]) (logid:4152e55e) Cleaned up 0 VM scheduled job entries 2025-06-29 04:03:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-3aaaee86]) (logid:82051bdf) VM Operation Thread Running 2025-06-29 04:03:48,614 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b48333e9]) (logid:5bb72efd) HA health check task is running... 2025-06-29 04:03:49,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d096a36]) (logid:938b3fe1) No inactive management server node found 2025-06-29 04:03:49,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d096a36]) (logid:938b3fe1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:50,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8444: Seq 7-8444: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:50,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8444 2025-06-29 04:03:50,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1808489]) (logid:dfa35c8e) No inactive management server node found 2025-06-29 04:03:50,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1808489]) (logid:dfa35c8e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00d41c51]) (logid:2c25cd93) No inactive management server node found 2025-06-29 04:03:52,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00d41c51]) (logid:2c25cd93) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:52,615 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f98a2212]) (logid:3f74f0ff) HA health check task is running... 2025-06-29 04:03:53,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4b4038a]) (logid:a803b06b) No inactive management server node found 2025-06-29 04:03:53,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4b4038a]) (logid:a803b06b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:55,278 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8445: Seq 7-8445: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:03:55,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8445 2025-06-29 04:03:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dd35084]) (logid:a86bfb73) No inactive management server node found 2025-06-29 04:03:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dd35084]) (logid:a86bfb73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:56,617 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-75f822b2]) (logid:6899c280) HA health check task is running... 2025-06-29 04:03:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b1043b0]) (logid:05264dc1) No inactive management server node found 2025-06-29 04:03:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b1043b0]) (logid:05264dc1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-424a09fd]) (logid:f605eb03) Enabled non-edge zones available for scan: 1 2025-06-29 04:03:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-424a09fd]) (logid:f605eb03) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:03:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-424a09fd]) (logid:f605eb03) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:03:58,111 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-6acd9e16]) (logid:ac589676) AutoScaling Monitor is running... 2025-06-29 04:03:58,122 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-75a83b64]) (logid:478a91e6) Starting to remove all volume_stats rows older than [Sat Jun 28 16:03:58 CEST 2025]. 2025-06-29 04:03:58,123 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-75a83b64]) (logid:478a91e6) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:03:58 CEST 2025]. 2025-06-29 04:03:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cced5e76]) (logid:89fc9ca5) Enabled non-edge zones available for scan: 1 2025-06-29 04:03:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-cced5e76]) (logid:89fc9ca5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:03:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28f4a51d]) (logid:678d16b3) No inactive management server node found 2025-06-29 04:03:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28f4a51d]) (logid:678d16b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:03:58,910 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-55f62a42]) (logid:9e0bbef2) Starting to remove all vm_stats rows older than [Sat Jun 28 16:03:58 CEST 2025]. 2025-06-29 04:03:58,935 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-55f62a42]) (logid:9e0bbef2) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:03:58 CEST 2025]. 2025-06-29 04:03:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7596bcb2]) (logid:3f2d149a) No inactive management server node found 2025-06-29 04:03:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7596bcb2]) (logid:3f2d149a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:00,430 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:04:00,432 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:04:00,434 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) Seq 2-4725683383995203706: Routed from 90520738888109 2025-06-29 04:04:00,437 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) Seq 1-4725683383995203706: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:04:00,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:04:00,439 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) Seq 2-3926857400090361985: Routed from 90520738888109 2025-06-29 04:04:00,484 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-61e10bc7]) (logid:03d2245b) Seq 2-3926857400090361985: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:04:00,618 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-507e0a63]) (logid:d2bc0dfd) HA health check task is running... 2025-06-29 04:04:01,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbf93b59]) (logid:33ce32e7) No inactive management server node found 2025-06-29 04:04:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbf93b59]) (logid:33ce32e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:02,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb5511f2]) (logid:dcaa08f2) No inactive management server node found 2025-06-29 04:04:02,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb5511f2]) (logid:dcaa08f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8084b765]) (logid:c69881f2) No inactive management server node found 2025-06-29 04:04:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8084b765]) (logid:c69881f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:04,619 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1f8e393f]) (logid:b9192559) HA health check task is running... 2025-06-29 04:04:05,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8446: Seq 7-8446: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:04:05,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8446 2025-06-29 04:04:05,317 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:04:05,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:04:05,323 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) Seq 2-4725683383995203707: Routed from 90520738888109 2025-06-29 04:04:05,378 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) Seq 1-4725683383995203707: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:04:05,416 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:04:05,417 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) Seq 2-3926857400090361986: Routed from 90520738888109 2025-06-29 04:04:05,518 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-746aba8e]) (logid:b2ca5967) Seq 2-3926857400090361986: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:04:05,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da67aa20]) (logid:29f6a0f2) No inactive management server node found 2025-06-29 04:04:05,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da67aa20]) (logid:29f6a0f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:06,403 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) StorageCollector is running... 2025-06-29 04:04:06,409 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:04:06,409 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:04:06,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:04:06,410 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 2-3938116399158788131: Routed from 90520738888109 2025-06-29 04:04:06,463 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 4-3938116399158788131: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:04:06,467 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:04:06,467 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:04:06,467 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:04:06,468 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 2-4725683383995203708: Routed from 90520738888109 2025-06-29 04:04:06,517 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 1-4725683383995203708: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:04:06,540 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:04:06,540 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:04:06,540 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:04:06,542 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 2-4725683383995203709: Routed from 90520738888109 2025-06-29 04:04:06,629 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 1-4725683383995203709: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:04:06,640 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:04:06,640 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:04:06,640 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:04:06,641 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 2-4725683383995203710: Routed from 90520738888109 2025-06-29 04:04:06,726 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 1-4725683383995203710: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:04:06,727 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:04:06,728 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:04:06,728 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:04:06,729 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 2-3926857400090361987: Routed from 90520738888109 2025-06-29 04:04:06,732 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-dc7cf053]) (logid:2410d721) Seq 2-3926857400090361987: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:04:07,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d58d6194]) (logid:160951bf) No inactive management server node found 2025-06-29 04:04:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d58d6194]) (logid:160951bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:08,621 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3650ea4d]) (logid:2985f346) HA health check task is running... 2025-06-29 04:04:08,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01b66c89]) (logid:253e665c) No inactive management server node found 2025-06-29 04:04:08,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01b66c89]) (logid:253e665c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:10,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25a0884e]) (logid:ef92cdfd) No inactive management server node found 2025-06-29 04:04:10,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25a0884e]) (logid:ef92cdfd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:11,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-234cf978]) (logid:10d429cd) No inactive management server node found 2025-06-29 04:04:11,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-234cf978]) (logid:10d429cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:12,622 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b2e7bb09]) (logid:aa90f31c) HA health check task is running... 2025-06-29 04:04:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9d2a1ce4]) (logid:8ad285c6) Found 0 routers to update status. 2025-06-29 04:04:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9d2a1ce4]) (logid:8ad285c6) Found 0 VPC's to update Redundant State. 2025-06-29 04:04:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9d2a1ce4]) (logid:8ad285c6) Found 0 networks to update RvR status. 2025-06-29 04:04:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63a73dc5]) (logid:db89db2c) No inactive management server node found 2025-06-29 04:04:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63a73dc5]) (logid:db89db2c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:14,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-340dc06d]) (logid:9eb652b4) No inactive management server node found 2025-06-29 04:04:14,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-340dc06d]) (logid:9eb652b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:15,279 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8447: Seq 7-8447: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:04:15,288 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8447 2025-06-29 04:04:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79f50bfd]) (logid:82625925) No inactive management server node found 2025-06-29 04:04:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79f50bfd]) (logid:82625925) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:16,623 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e54a3e55]) (logid:4c9c5d06) HA health check task is running... 2025-06-29 04:04:17,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7e14909]) (logid:d1df2793) No inactive management server node found 2025-06-29 04:04:17,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7e14909]) (logid:d1df2793) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:19,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3716823d]) (logid:cc066205) No inactive management server node found 2025-06-29 04:04:19,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3716823d]) (logid:cc066205) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:20,624 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-58b56474]) (logid:a6862d88) HA health check task is running... 2025-06-29 04:04:20,869 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49a949cf]) (logid:e93fc5c7) No inactive management server node found 2025-06-29 04:04:20,869 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49a949cf]) (logid:e93fc5c7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:22,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7787911b]) (logid:5a34d237) No inactive management server node found 2025-06-29 04:04:22,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7787911b]) (logid:5a34d237) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:23,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5922928a]) (logid:e9909a09) No inactive management server node found 2025-06-29 04:04:23,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5922928a]) (logid:e9909a09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:24,626 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-39c3596f]) (logid:ad0b51f5) HA health check task is running... 2025-06-29 04:04:25,280 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8448: Seq 7-8448: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:04:25,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8448 2025-06-29 04:04:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcf86fd5]) (logid:977e8d5f) No inactive management server node found 2025-06-29 04:04:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcf86fd5]) (logid:977e8d5f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:26,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00ed0f5d]) (logid:3184a1ce) No inactive management server node found 2025-06-29 04:04:26,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00ed0f5d]) (logid:3184a1ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9f55eaa8]) (logid:33fd1b87) Enabled non-edge zones available for scan: 1 2025-06-29 04:04:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9f55eaa8]) (logid:33fd1b87) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:04:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9f55eaa8]) (logid:33fd1b87) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:04:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-87203ee8]) (logid:0b0d4fd4) Enabled non-edge zones available for scan: 1 2025-06-29 04:04:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-87203ee8]) (logid:0b0d4fd4) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:04:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-172e7551]) (logid:603bf290) No inactive management server node found 2025-06-29 04:04:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-172e7551]) (logid:603bf290) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:28,627 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-17216564]) (logid:d5912742) HA health check task is running... 2025-06-29 04:04:29,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c18cb68]) (logid:784422bd) No inactive management server node found 2025-06-29 04:04:29,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c18cb68]) (logid:784422bd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:31,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b74bad33]) (logid:fc9bef3b) No inactive management server node found 2025-06-29 04:04:31,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b74bad33]) (logid:fc9bef3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:32,628 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2c35f7b9]) (logid:b2ee5861) HA health check task is running... 2025-06-29 04:04:32,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b633ef3]) (logid:c0114666) No inactive management server node found 2025-06-29 04:04:32,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b633ef3]) (logid:c0114666) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6cc32c30]) (logid:4217c1a2) No inactive management server node found 2025-06-29 04:04:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6cc32c30]) (logid:4217c1a2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:35,280 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8449: Seq 7-8449: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:04:35,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8449 2025-06-29 04:04:35,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e2927c2]) (logid:c9457207) No inactive management server node found 2025-06-29 04:04:35,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e2927c2]) (logid:c9457207) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:36,630 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a6c30821]) (logid:fea46b27) HA health check task is running... 2025-06-29 04:04:37,372 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d71bbe3]) (logid:df194678) No inactive management server node found 2025-06-29 04:04:37,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d71bbe3]) (logid:df194678) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:38,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ceb5657d]) (logid:42e881bd) No inactive management server node found 2025-06-29 04:04:38,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ceb5657d]) (logid:42e881bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11ab2ced]) (logid:089d7873) No inactive management server node found 2025-06-29 04:04:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11ab2ced]) (logid:089d7873) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:40,631 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b39a4be7]) (logid:e603ea58) HA health check task is running... 2025-06-29 04:04:41,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6a4731f]) (logid:bf855cf6) No inactive management server node found 2025-06-29 04:04:41,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6a4731f]) (logid:bf855cf6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-bc3e5d90]) (logid:c010b13c) DbCollector is running... 2025-06-29 04:04:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1feaa9b9]) (logid:01314817) Found 0 routers to update status. 2025-06-29 04:04:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1feaa9b9]) (logid:01314817) Found 0 VPC's to update Redundant State. 2025-06-29 04:04:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1feaa9b9]) (logid:01314817) Found 0 networks to update RvR status. 2025-06-29 04:04:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c09c8c9c]) (logid:fa39d985) No inactive management server node found 2025-06-29 04:04:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c09c8c9c]) (logid:fa39d985) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) ManagementServerCollector is running... 2025-06-29 04:04:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) DB properties were already loaded 2025-06-29 04:04:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:04:44,059 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Execution of process [4066009] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:04:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Exit value of process [4066009] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:04:44,059 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Process [4066009] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:04:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) usage install: null 2025-06-29 04:04:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Sessions active: 0 2025-06-29 04:04:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:04:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066012] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:04:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:04:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066017] for command [/bin/bash -c uname -r ]. 2025-06-29 04:04:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:04:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066018] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:04:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) system memory from /proc: 4887515136 2025-06-29 04:04:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:04:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066023] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:04:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) free memory from /proc: 302841856 2025-06-29 04:04:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:04:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066028] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:04:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) used memory from /proc: 1069504 2025-06-29 04:04:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:04:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066029] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:04:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:04:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066030] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:04:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:04:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066031] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:04:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:04:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066038] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:04:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:04:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Successfully executed process [4066039] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail buffersdirect.capacity, 318249 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail buffersdirect.count, 92 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail buffersdirect.used, 318249 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail buffersmapped.capacity, 0 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail buffersmapped.count, 0 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail buffersmapped.used, 0 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail jvmuptime, 1743832 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memoryheap.committed, 500695040 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memoryheap.init, 77594624 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memoryheap.usage, 0.12183467505627103 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorynon-heap.committed, 194183168 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorynon-heap.max, -1 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorynon-heap.usage, -1.91352064E8 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorynon-heap.used, 191352064 2025-06-29 04:04:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10635288922953878 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2066299113540174 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.Compressed-Class-Space.usage, 0.012370340526103973 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.Metaspace.usage, 0.992938192077579 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.PS-Eden-Space.usage, 0.04774984550321131 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorytotal.committed, 694878208 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorytotal.init, 85262336 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorytotal.max, 1908932607 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail memorytotal.used, 423926248 2025-06-29 04:04:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail threadsdeadlocks, [] 2025-06-29 04:04:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail threadsnew.count, 0 2025-06-29 04:04:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:04:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-d7b71033]) (logid:db4d54a4) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:04:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":268759176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1743762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":423287160,"totalCommitted":694878208,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":232574184,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302841856,"systemMemoryUsed":1069504,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.43,0.39],"systemCyclesUsage":[66406131,53828677,4589287761],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:04:44,133 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 57, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:04:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":268759176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1743762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":423287160,"totalCommitted":694878208,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":232574184,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302841856,"systemMemoryUsed":1069504,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.43,0.39],"systemCyclesUsage":[66406131,53828677,4589287761],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:04:44,133 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:04:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:04:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 57, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:04:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":268759176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1743762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":423287160,"totalCommitted":694878208,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":232574184,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302841856,"systemMemoryUsed":1069504,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.43,0.39],"systemCyclesUsage":[66406131,53828677,4589287761],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:04:44,142 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-15:[ctx-5a77c615]) (logid:fcb19dd3) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:04:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":268759176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1743762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":423287160,"totalCommitted":694878208,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":232574184,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302841856,"systemMemoryUsed":1069504,"systemMemoryVirtualSize":6218358784,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.43,0.39],"systemCyclesUsage":[66406131,53828677,4589287761],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:04:44,633 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8cf443f8]) (logid:c659179e) HA health check task is running... 2025-06-29 04:04:44,872 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01d3b4cd]) (logid:0dc8d2fa) No inactive management server node found 2025-06-29 04:04:44,872 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01d3b4cd]) (logid:0dc8d2fa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:45,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8450: Seq 7-8450: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:04:45,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8450 2025-06-29 04:04:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:04:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:04:46,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98cacab9]) (logid:7e117428) No inactive management server node found 2025-06-29 04:04:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98cacab9]) (logid:7e117428) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:04:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:04:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:04:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:04:47,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:04:47,069 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:04:47,070 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:04:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:04:47,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58f5dd99]) (logid:e939e694) No inactive management server node found 2025-06-29 04:04:47,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58f5dd99]) (logid:e939e694) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:48,069 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7cef737f]) (logid:e9a75aaf) ClusterDRS.poll is being called at 2025-06-29T02:05:00+0000 2025-06-29 04:04:48,077 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7cef737f]) (logid:e9a75aaf) Removed 0 old drs migration plans 2025-06-29 04:04:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-eabbd91d]) (logid:39afd95b) VM scheduler.poll is being called at 2025-06-29T02:05:00+0000 2025-06-29 04:04:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-eabbd91d]) (logid:39afd95b) Got 0 scheduled jobs to be executed at 2025-06-29T02:05:00+0000 2025-06-29 04:04:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-eabbd91d]) (logid:39afd95b) Cleaned up 0 VM scheduled job entries 2025-06-29 04:04:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-0997690d]) (logid:1ac53ef2) VM Operation Thread Running 2025-06-29 04:04:48,634 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-bf63e714]) (logid:42c4e172) HA health check task is running... 2025-06-29 04:04:49,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bee1b45]) (logid:70870145) No inactive management server node found 2025-06-29 04:04:49,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bee1b45]) (logid:70870145) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:50,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24d86bfc]) (logid:44ab4d65) No inactive management server node found 2025-06-29 04:04:50,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24d86bfc]) (logid:44ab4d65) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:52,371 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2514d9e8]) (logid:af3fd09d) No inactive management server node found 2025-06-29 04:04:52,372 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2514d9e8]) (logid:af3fd09d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:52,635 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-665301b4]) (logid:f391af6a) HA health check task is running... 2025-06-29 04:04:53,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d3e86e4]) (logid:1b4fb962) No inactive management server node found 2025-06-29 04:04:53,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d3e86e4]) (logid:1b4fb962) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:55,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8452: Seq 7-8452: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:04:55,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8452 2025-06-29 04:04:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a58c003]) (logid:1d1a3f1f) No inactive management server node found 2025-06-29 04:04:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a58c003]) (logid:1d1a3f1f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:56,637 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ea929441]) (logid:2c269bda) HA health check task is running... 2025-06-29 04:04:56,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2c4d380]) (logid:7f56d58c) No inactive management server node found 2025-06-29 04:04:56,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2c4d380]) (logid:7f56d58c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8ff720c3]) (logid:87f73f37) Enabled non-edge zones available for scan: 1 2025-06-29 04:04:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8ff720c3]) (logid:87f73f37) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:04:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8ff720c3]) (logid:87f73f37) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:04:58,112 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-deb3cbd2]) (logid:8c72316f) AutoScaling Monitor is running... 2025-06-29 04:04:58,124 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-eddad83d]) (logid:9861ec39) Starting to remove all volume_stats rows older than [Sat Jun 28 16:04:58 CEST 2025]. 2025-06-29 04:04:58,125 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-eddad83d]) (logid:9861ec39) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:04:58 CEST 2025]. 2025-06-29 04:04:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a3dbfe48]) (logid:7ca4d187) Enabled non-edge zones available for scan: 1 2025-06-29 04:04:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a3dbfe48]) (logid:7ca4d187) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:04:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fed8f6d]) (logid:4f64ee00) No inactive management server node found 2025-06-29 04:04:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fed8f6d]) (logid:4f64ee00) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:04:58,936 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-e10bf2e8]) (logid:61004798) Starting to remove all vm_stats rows older than [Sat Jun 28 16:04:58 CEST 2025]. 2025-06-29 04:04:58,959 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-e10bf2e8]) (logid:61004798) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:04:58 CEST 2025]. 2025-06-29 04:04:59,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38d63b65]) (logid:b247989c) No inactive management server node found 2025-06-29 04:04:59,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38d63b65]) (logid:b247989c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:00,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8453: Seq 7-8453: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:00,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8453 2025-06-29 04:05:00,488 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:05:00,490 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:05:00,492 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) Seq 2-4725683383995203711: Routed from 90520738888109 2025-06-29 04:05:00,536 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) Seq 1-4725683383995203711: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:05:00,537 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:05:00,538 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) Seq 2-3926857400090361988: Routed from 90520738888109 2025-06-29 04:05:00,583 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-7fdd6930]) (logid:14f0511c) Seq 2-3926857400090361988: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:05:00,638 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-12ed6c08]) (logid:d984a1cc) HA health check task is running... 2025-06-29 04:05:01,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44615d30]) (logid:3a7c2211) No inactive management server node found 2025-06-29 04:05:01,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44615d30]) (logid:3a7c2211) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:02,870 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4c5f142]) (logid:1db256ca) No inactive management server node found 2025-06-29 04:05:02,870 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4c5f142]) (logid:1db256ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:04,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60d0bb3d]) (logid:1067ec74) No inactive management server node found 2025-06-29 04:05:04,370 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60d0bb3d]) (logid:1067ec74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:04,640 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-519811f4]) (logid:7c5e6eef) HA health check task is running... 2025-06-29 04:05:05,579 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:05:05,582 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:05:05,582 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) Seq 2-4725683383995203712: Routed from 90520738888109 2025-06-29 04:05:05,634 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) Seq 1-4725683383995203712: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:05:05,681 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:05:05,682 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) Seq 2-3926857400090361989: Routed from 90520738888109 2025-06-29 04:05:05,768 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e4fb80a6]) (logid:881294da) Seq 2-3926857400090361989: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:05:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc81aa7a]) (logid:bf967af8) No inactive management server node found 2025-06-29 04:05:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc81aa7a]) (logid:bf967af8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:06,732 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) StorageCollector is running... 2025-06-29 04:05:06,738 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:05:06,738 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:05:06,738 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:05:06,739 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 2-3938116399158788132: Routed from 90520738888109 2025-06-29 04:05:06,817 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 4-3938116399158788132: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:05:06,821 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:05:06,821 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:05:06,821 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:05:06,822 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 2-4725683383995203713: Routed from 90520738888109 2025-06-29 04:05:06,869 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 1-4725683383995203713: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:05:06,891 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:05:06,891 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:05:06,891 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:05:06,892 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 2-4725683383995203714: Routed from 90520738888109 2025-06-29 04:05:06,980 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 1-4725683383995203714: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:05:06,991 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:05:06,991 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:05:06,991 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:05:06,992 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 2-4725683383995203715: Routed from 90520738888109 2025-06-29 04:05:07,078 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 1-4725683383995203715: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:05:07,080 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:05:07,080 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:05:07,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:05:07,081 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 2-3926857400090361990: Routed from 90520738888109 2025-06-29 04:05:07,085 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4db314c4]) (logid:b79b6f47) Seq 2-3926857400090361990: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:05:07,370 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d3b7f36]) (logid:73943a00) No inactive management server node found 2025-06-29 04:05:07,371 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d3b7f36]) (logid:73943a00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:08,641 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-6b9d819c]) (logid:f6d8a8ad) HA health check task is running... 2025-06-29 04:05:08,873 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21d987bb]) (logid:33a7a7ec) No inactive management server node found 2025-06-29 04:05:08,873 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21d987bb]) (logid:33a7a7ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:10,283 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8454: Seq 7-8454: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:10,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8454 2025-06-29 04:05:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29d14a02]) (logid:711cb23b) No inactive management server node found 2025-06-29 04:05:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29d14a02]) (logid:711cb23b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:11,871 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8649228f]) (logid:8f233f39) No inactive management server node found 2025-06-29 04:05:11,871 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8649228f]) (logid:8f233f39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:12,642 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f813afed]) (logid:7eadc69f) HA health check task is running... 2025-06-29 04:05:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c600f755]) (logid:993c4068) Found 0 routers to update status. 2025-06-29 04:05:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c600f755]) (logid:993c4068) Found 0 VPC's to update Redundant State. 2025-06-29 04:05:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c600f755]) (logid:993c4068) Found 0 networks to update RvR status. 2025-06-29 04:05:15,154 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36700e77]) (logid:ddb1e82a) No inactive management server node found 2025-06-29 04:05:15,154 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36700e77]) (logid:ddb1e82a) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:15,155 DEBUG [c.c.c.ClusterManagerImpl$4] (Cluster-Heartbeat-1:[ctx-36700e77]) (logid:ddb1e82a) Management server heartbeat takes too long to finish. profiler: Done. Duration: 1807ms, profilerHeartbeatUpdate: Done. Duration: 1803ms, profilerPeerScan: Done. Duration: 3ms 2025-06-29 04:05:15,179 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0e5cf2b]) (logid:492bea47) No inactive management server node found 2025-06-29 04:05:15,179 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0e5cf2b]) (logid:492bea47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:16,644 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-dac63c4e]) (logid:d3954908) HA health check task is running... 2025-06-29 04:05:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9c060f0d]) (logid:99cbd548) Resetting hosts suitable for reconnect 2025-06-29 04:05:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9c060f0d]) (logid:99cbd548) Completed resetting hosts suitable for reconnect 2025-06-29 04:05:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9c060f0d]) (logid:99cbd548) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:05:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9c060f0d]) (logid:99cbd548) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:05:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9c060f0d]) (logid:99cbd548) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:05:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9c060f0d]) (logid:99cbd548) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:05:20,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8455: Seq 7-8455: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:20,645 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-dbabb7ca]) (logid:e62f87e9) HA health check task is running... 2025-06-29 04:05:24,647 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5c0251f9]) (logid:503a4119) HA health check task is running... 2025-06-29 04:05:25,896 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93ffc843]) (logid:9c11c598) No inactive management server node found 2025-06-29 04:05:25,896 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93ffc843]) (logid:9c11c598) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:05:25,896 DEBUG [c.c.c.ClusterManagerImpl$4] (Cluster-Heartbeat-1:[ctx-93ffc843]) (logid:9c11c598) Management server heartbeat takes too long to finish. profiler: Done. Duration: 9548ms, profilerHeartbeatUpdate: Done. Duration: 9546ms, profilerPeerScan: Done. Duration: 1ms 2025-06-29 04:05:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-be25cad9]) (logid:5225e907) Enabled non-edge zones available for scan: 1 2025-06-29 04:05:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-be25cad9]) (logid:5225e907) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:05:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-be25cad9]) (logid:5225e907) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:05:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1e1445a1]) (logid:f7eed89e) Enabled non-edge zones available for scan: 1 2025-06-29 04:05:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1e1445a1]) (logid:f7eed89e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:05:28,648 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d9f17733]) (logid:38a3d27e) HA health check task is running... 2025-06-29 04:05:30,242 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8456: Seq 7-8456: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:32,649 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-75c92cf9]) (logid:b8d59049) HA health check task is running... 2025-06-29 04:05:36,651 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-efda4954]) (logid:f7cf1d34) HA health check task is running... 2025-06-29 04:05:40,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8457: Seq 7-8457: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:42,197 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e689f5cd]) (logid:a078cd54) HA health check task is running... 2025-06-29 04:05:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-7baea973]) (logid:c4a3405b) DbCollector is running... 2025-06-29 04:05:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-615ee883]) (logid:681a3b9a) Found 0 routers to update status. 2025-06-29 04:05:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-615ee883]) (logid:681a3b9a) Found 0 VPC's to update Redundant State. 2025-06-29 04:05:43,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-615ee883]) (logid:681a3b9a) Found 0 networks to update RvR status. 2025-06-29 04:05:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) ManagementServerCollector is running... 2025-06-29 04:05:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) DB properties were already loaded 2025-06-29 04:05:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:05:44,163 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Execution of process [4066050] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:05:44,163 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Exit value of process [4066050] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:05:44,164 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Process [4066050] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:05:44,164 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) usage install: null 2025-06-29 04:05:44,164 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Sessions active: 0 2025-06-29 04:05:44,167 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:05:45,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8458: Seq 7-8458: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:05:45,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:05:46,188 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066053] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:05:46,189 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:05:46,198 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1a8163cf]) (logid:795bb413) HA health check task is running... 2025-06-29 04:05:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7cc79a2a]) (logid:a4654095) VM scheduler.poll is being called at 2025-06-29T02:06:00+0000 2025-06-29 04:05:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-87250888]) (logid:4174a9e0) VM Operation Thread Running 2025-06-29 04:05:48,345 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066058] for command [/bin/bash -c uname -r ]. 2025-06-29 04:05:48,345 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:05:48,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:05:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3b996d5f]) (logid:32eaa743) ClusterDRS.poll is being called at 2025-06-29T02:06:00+0000 2025-06-29 04:05:48,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:05:48,355 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:05:48,358 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:05:48,377 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066059] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:05:48,377 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) system memory from /proc: 4887515136 2025-06-29 04:05:48,377 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:05:48,427 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066064] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:05:48,427 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) free memory from /proc: 296349696 2025-06-29 04:05:48,427 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:05:48,440 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066069] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:05:48,440 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) used memory from /proc: 1069760 2025-06-29 04:05:48,440 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:05:48,446 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066070] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:05:48,447 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:05:48,460 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066071] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:05:48,460 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:05:48,474 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066072] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:05:48,475 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:05:48,480 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066079] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:05:48,480 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:05:48,487 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Successfully executed process [4066080] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:05:52,360 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-34036ee7]) (logid:7072ffa9) HA health check task is running... 2025-06-29 04:05:54,294 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8455 2025-06-29 04:05:55,454 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8460: Seq 7-8460: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:05:56,284 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:05:58,244 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-6116f081]) (logid:8f4c621d) AutoScaling Monitor is running... 2025-06-29 04:06:02,170 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-6e469a05]) (logid:2d8c2346) HA health check task is running... 2025-06-29 04:06:02,171 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:06:02,175 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:06:06,113 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8461: Seq 7-8461: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:10,040 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail buffersdirect.capacity, 317346 2025-06-29 04:06:10,040 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail buffersdirect.count, 92 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail buffersdirect.used, 317346 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail buffersmapped.capacity, 0 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail buffersmapped.count, 0 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail buffersmapped.used, 0 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:06:10,041 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail jvmuptime, 1829748 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memoryheap.committed, 500695040 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memoryheap.init, 77594624 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memoryheap.usage, 0.13692288921286005 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorynon-heap.committed, 194314240 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorynon-heap.max, -1 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorynon-heap.usage, -1.91850688E8 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorynon-heap.used, 191850688 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3042324561403509 2025-06-29 04:06:10,045 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10743593375099973 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.209010605858633 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.Compressed-Class-Space.usage, 0.012370854616165161 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.Metaspace.usage, 0.9934593827461048 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.PS-Eden-Space.usage, 0.09273478851194406 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorytotal.committed, 695009280 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorytotal.init, 85262336 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorytotal.max, 1908932607 2025-06-29 04:06:10,046 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail memorytotal.used, 453458088 2025-06-29 04:06:10,053 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail threadsdeadlocks, [] 2025-06-29 04:06:10,055 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail threadsnew.count, 0 2025-06-29 04:06:10,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:06:12,006 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:06:14,047 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:06:16,013 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-836ef256]) (logid:4477d9a7) HA health check task is running... 2025-06-29 04:06:16,015 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8462: Seq 7-8462: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:17,985 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7cc79a2a]) (logid:a4654095) Got 0 scheduled jobs to be executed at 2025-06-29T02:06:00+0000 2025-06-29 04:06:17,985 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) StorageCollector is running... 2025-06-29 04:06:17,990 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8456 2025-06-29 04:06:17,994 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0f529865]) (logid:bdb59fa8) Found 0 routers to update status. 2025-06-29 04:06:17,994 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:06:17,995 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0f529865]) (logid:bdb59fa8) Found 0 VPC's to update Redundant State. 2025-06-29 04:06:17,997 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-3b996d5f]) (logid:32eaa743) Removed 0 old drs migration plans 2025-06-29 04:06:17,999 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-7cc79a2a]) (logid:a4654095) Cleaned up 0 VM scheduled job entries 2025-06-29 04:06:18,003 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14dabfac]) (logid:23607712) No inactive management server node found 2025-06-29 04:06:18,003 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14dabfac]) (logid:23607712) Peer scan is finished. profiler: Done. Duration: 3958ms , profilerQueryActiveList: Done. Duration: 3941ms, , profilerSyncClusterInfo: Done. Duration: 4ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 11ms 2025-06-29 04:06:18,003 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14dabfac]) (logid:23607712) Peer scan takes too long to finish. profiler: Done. Duration: 3958ms , profilerQueryActiveList: Done. Duration: 3941ms, , profilerSyncClusterInfo: Done. Duration: 4ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 11ms 2025-06-29 04:06:18,003 DEBUG [c.c.c.ClusterManagerImpl$4] (Cluster-Heartbeat-1:[ctx-14dabfac]) (logid:23607712) Management server heartbeat takes too long to finish. profiler: Done. Duration: 52106ms, profilerHeartbeatUpdate: Done. Duration: 48148ms, profilerPeerScan: Done. Duration: 3958ms 2025-06-29 04:06:18,004 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:06:18,016 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0f529865]) (logid:bdb59fa8) Found 0 networks to update RvR status. 2025-06-29 04:06:18,023 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8457 2025-06-29 04:06:18,025 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-24bebee5]) (logid:0cea859e) Starting to remove all volume_stats rows older than [Sat Jun 28 16:06:08 CEST 2025]. 2025-06-29 04:06:18,025 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-ba64611d]) (logid:010eda0a) Starting to remove all vm_stats rows older than [Sat Jun 28 16:06:10 CEST 2025]. 2025-06-29 04:06:18,027 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) Seq 2-4725683383995203716: Routed from 90520738888109 2025-06-29 04:06:18,027 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-2527a166]) (logid:71259353) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:05:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":245992320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1803867,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.68,"totalInit":85262336,"totalUsed":446426744,"totalCommitted":694943744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":261376568,"heapMemoryTotal":1908932608,"threadsBlockedCount":3,"threadsDaemonCount":17,"threadsRunnableCount":17,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":96,"systemMemoryTotal":4887515136,"systemMemoryFree":296349696,"systemMemoryUsed":1069760,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.68,0.73,0.49],"systemCyclesUsage":[66406988,53828984,4589309816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:06:18,028 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 59, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:05:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":245992320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1803867,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.68,"totalInit":85262336,"totalUsed":446426744,"totalCommitted":694943744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":261376568,"heapMemoryTotal":1908932608,"threadsBlockedCount":3,"threadsDaemonCount":17,"threadsRunnableCount":17,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":96,"systemMemoryTotal":4887515136,"systemMemoryFree":296349696,"systemMemoryUsed":1069760,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.68,0.73,0.49],"systemCyclesUsage":[66406988,53828984,4589309816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:06:18,028 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:06:18,035 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) Seq 1-4725683383995203716: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:06:18,036 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8458 2025-06-29 04:06:18,041 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 12 ms 2025-06-29 04:06:18,042 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 13ms. agent: 0, pdu seq: 59, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:05:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":245992320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1803867,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.68,"totalInit":85262336,"totalUsed":446426744,"totalCommitted":694943744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":261376568,"heapMemoryTotal":1908932608,"threadsBlockedCount":3,"threadsDaemonCount":17,"threadsRunnableCount":17,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":96,"systemMemoryTotal":4887515136,"systemMemoryFree":296349696,"systemMemoryUsed":1069760,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.68,0.73,0.49],"systemCyclesUsage":[66406988,53828984,4589309816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:06:18,043 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-24bebee5]) (logid:0cea859e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:06:08 CEST 2025]. 2025-06-29 04:06:18,046 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-16:[ctx-34c77fcf]) (logid:99db79aa) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:05:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":245992320,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1803867,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.68,"totalInit":85262336,"totalUsed":446426744,"totalCommitted":694943744,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":261376568,"heapMemoryTotal":1908932608,"threadsBlockedCount":3,"threadsDaemonCount":17,"threadsRunnableCount":17,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":96,"systemMemoryTotal":4887515136,"systemMemoryFree":296349696,"systemMemoryUsed":1069760,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.68,0.73,0.49],"systemCyclesUsage":[66406988,53828984,4589309816],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:06:18,049 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8460 2025-06-29 04:06:18,054 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-494baf12]) (logid:e6298394) Enabled non-edge zones available for scan: 1 2025-06-29 04:06:18,055 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:06:18,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8462 2025-06-29 04:06:18,058 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:06:18,060 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) Seq 2-3926857400090361991: Routed from 90520738888109 2025-06-29 04:06:18,062 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) Seq 2-4725683383995203717: Routed from 90520738888109 2025-06-29 04:06:18,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8461 2025-06-29 04:06:18,070 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67a81583]) (logid:1af40c72) Enabled non-edge zones available for scan: 1 2025-06-29 04:06:18,080 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4acad8d3]) (logid:75b3caef) No inactive management server node found 2025-06-29 04:06:18,080 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4acad8d3]) (logid:75b3caef) Peer scan is finished. profiler: Done. Duration: 61ms , profilerQueryActiveList: Done. Duration: 20ms, , profilerSyncClusterInfo: Done. Duration: 34ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 6ms 2025-06-29 04:06:18,090 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:06:18,090 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:06:18,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:06:18,092 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-494baf12]) (logid:e6298394) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:06:18,093 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 2-3938116399158788133: Routed from 90520738888109 2025-06-29 04:06:18,096 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5008f7ac]) (logid:353c5d31) No inactive management server node found 2025-06-29 04:06:18,096 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5008f7ac]) (logid:353c5d31) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-67a81583]) (logid:1af40c72) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:06:18,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67a81583]) (logid:1af40c72) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:06:18,102 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-ba64611d]) (logid:010eda0a) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:06:10 CEST 2025]. 2025-06-29 04:06:18,104 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c742ef57]) (logid:159aab5d) Seq 2-3926857400090361991: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:06:18,105 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62bfe07c]) (logid:5ec04db9) No inactive management server node found 2025-06-29 04:06:18,105 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62bfe07c]) (logid:5ec04db9) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:06:18,110 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a36277c]) (logid:7894b819) No inactive management server node found 2025-06-29 04:06:18,110 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a36277c]) (logid:7894b819) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,116 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9d821e7]) (logid:066b1c62) No inactive management server node found 2025-06-29 04:06:18,116 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9d821e7]) (logid:066b1c62) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:06:18,121 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50403f15]) (logid:addc73a2) No inactive management server node found 2025-06-29 04:06:18,121 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50403f15]) (logid:addc73a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,126 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1820232b]) (logid:6934f0d1) No inactive management server node found 2025-06-29 04:06:18,126 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1820232b]) (logid:6934f0d1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:06:18,131 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96720d6c]) (logid:1aef3b00) No inactive management server node found 2025-06-29 04:06:18,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96720d6c]) (logid:1aef3b00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,135 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55f78b3d]) (logid:abd92c75) No inactive management server node found 2025-06-29 04:06:18,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55f78b3d]) (logid:abd92c75) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,138 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b647be2]) (logid:9934f268) No inactive management server node found 2025-06-29 04:06:18,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b647be2]) (logid:9934f268) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,141 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cd4858a]) (logid:3f53d426) No inactive management server node found 2025-06-29 04:06:18,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cd4858a]) (logid:3f53d426) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,145 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f19165f]) (logid:b77006bd) No inactive management server node found 2025-06-29 04:06:18,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f19165f]) (logid:b77006bd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,148 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15b9c10b]) (logid:a4307f2c) No inactive management server node found 2025-06-29 04:06:18,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15b9c10b]) (logid:a4307f2c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,153 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-065a999d]) (logid:c74ef204) No inactive management server node found 2025-06-29 04:06:18,153 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-065a999d]) (logid:c74ef204) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,158 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f39d7c6e]) (logid:e2528494) No inactive management server node found 2025-06-29 04:06:18,158 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f39d7c6e]) (logid:e2528494) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,162 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06fa7673]) (logid:f1fc04b6) No inactive management server node found 2025-06-29 04:06:18,162 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06fa7673]) (logid:f1fc04b6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,163 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 4-3938116399158788133: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:06:18,166 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3c10695]) (logid:a3288f59) No inactive management server node found 2025-06-29 04:06:18,166 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3c10695]) (logid:a3288f59) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,167 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) Seq 1-4725683383995203717: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:06:18,170 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14914358]) (logid:00fbd3b1) No inactive management server node found 2025-06-29 04:06:18,170 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14914358]) (logid:00fbd3b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,171 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:06:18,171 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:06:18,171 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:06:18,173 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 2-4725683383995203718: Routed from 90520738888109 2025-06-29 04:06:18,175 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bf6c994]) (logid:6be866fd) No inactive management server node found 2025-06-29 04:06:18,175 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bf6c994]) (logid:6be866fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,179 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d265ac26]) (logid:119cc950) No inactive management server node found 2025-06-29 04:06:18,179 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d265ac26]) (logid:119cc950) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,184 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22bd8abc]) (logid:777211d3) No inactive management server node found 2025-06-29 04:06:18,184 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22bd8abc]) (logid:777211d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,189 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03932568]) (logid:174fe9de) No inactive management server node found 2025-06-29 04:06:18,189 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03932568]) (logid:174fe9de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,194 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9de045aa]) (logid:2f02fd8b) No inactive management server node found 2025-06-29 04:06:18,194 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9de045aa]) (logid:2f02fd8b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:06:18,199 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3af44873]) (logid:2b9d5eb0) No inactive management server node found 2025-06-29 04:06:18,199 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3af44873]) (logid:2b9d5eb0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,203 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3561eb81]) (logid:62ddfbc1) No inactive management server node found 2025-06-29 04:06:18,203 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3561eb81]) (logid:62ddfbc1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,208 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9487eefb]) (logid:4df78bb6) No inactive management server node found 2025-06-29 04:06:18,208 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9487eefb]) (logid:4df78bb6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:06:18,211 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) Seq 2-3926857400090361992: Routed from 90520738888109 2025-06-29 04:06:18,212 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b21e0d4]) (logid:9c86edaa) No inactive management server node found 2025-06-29 04:06:18,212 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b21e0d4]) (logid:9c86edaa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,216 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36a7b667]) (logid:feecce86) No inactive management server node found 2025-06-29 04:06:18,216 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36a7b667]) (logid:feecce86) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,220 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e27261f0]) (logid:047672c6) No inactive management server node found 2025-06-29 04:06:18,221 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e27261f0]) (logid:047672c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,225 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3caf5f04]) (logid:55e702b9) No inactive management server node found 2025-06-29 04:06:18,225 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3caf5f04]) (logid:55e702b9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,228 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c85b9654]) (logid:581040d0) No inactive management server node found 2025-06-29 04:06:18,228 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c85b9654]) (logid:581040d0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,231 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2adc60d5]) (logid:832bee07) No inactive management server node found 2025-06-29 04:06:18,231 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2adc60d5]) (logid:832bee07) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,234 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edef3624]) (logid:aaf0d041) No inactive management server node found 2025-06-29 04:06:18,235 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edef3624]) (logid:aaf0d041) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,237 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4d1441e]) (logid:b67c4bc2) No inactive management server node found 2025-06-29 04:06:18,237 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4d1441e]) (logid:b67c4bc2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,240 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ba4a50c]) (logid:d06fff24) No inactive management server node found 2025-06-29 04:06:18,240 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ba4a50c]) (logid:d06fff24) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,243 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9008112]) (logid:1160b8f7) No inactive management server node found 2025-06-29 04:06:18,243 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9008112]) (logid:1160b8f7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,246 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-feb48715]) (logid:12c64e5e) No inactive management server node found 2025-06-29 04:06:18,246 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-feb48715]) (logid:12c64e5e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,249 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bdb5977]) (logid:3a66c0da) No inactive management server node found 2025-06-29 04:06:18,249 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bdb5977]) (logid:3a66c0da) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,252 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee54c559]) (logid:3c7b8f24) No inactive management server node found 2025-06-29 04:06:18,252 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee54c559]) (logid:3c7b8f24) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,254 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b8c77df]) (logid:3c8d58d4) No inactive management server node found 2025-06-29 04:06:18,254 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b8c77df]) (logid:3c8d58d4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:18,262 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 1-4725683383995203718: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:06:18,274 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:06:18,274 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:06:18,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:06:18,275 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 2-4725683383995203719: Routed from 90520738888109 2025-06-29 04:06:18,297 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-269e8c48]) (logid:c300e11a) Seq 2-3926857400090361992: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:06:18,362 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 1-4725683383995203719: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:06:18,372 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:06:18,372 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:06:18,372 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:06:18,373 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 2-4725683383995203720: Routed from 90520738888109 2025-06-29 04:06:18,458 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 1-4725683383995203720: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:06:18,460 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:06:18,460 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:06:18,460 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:06:18,461 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 2-3926857400090361993: Routed from 90520738888109 2025-06-29 04:06:18,465 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ed1e29ce]) (logid:15da08d7) Seq 2-3926857400090361993: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:06:19,474 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4fbe2de]) (logid:7c36be21) No inactive management server node found 2025-06-29 04:06:19,474 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4fbe2de]) (logid:7c36be21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb900809]) (logid:b08be395) No inactive management server node found 2025-06-29 04:06:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb900809]) (logid:b08be395) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:22,025 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b03dde57]) (logid:9ce858ff) HA health check task is running... 2025-06-29 04:06:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fb120ec]) (logid:a91ae541) No inactive management server node found 2025-06-29 04:06:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fb120ec]) (logid:a91ae541) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a46bb79a]) (logid:bd2a9bb1) No inactive management server node found 2025-06-29 04:06:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a46bb79a]) (logid:bd2a9bb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:25,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8463: Seq 7-8463: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:25,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8463 2025-06-29 04:06:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06151257]) (logid:8ea03894) No inactive management server node found 2025-06-29 04:06:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06151257]) (logid:8ea03894) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:26,027 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-05a0b223]) (logid:a01df394) HA health check task is running... 2025-06-29 04:06:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ee27bd5]) (logid:63bcd8b5) No inactive management server node found 2025-06-29 04:06:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ee27bd5]) (logid:63bcd8b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a18f48f5]) (logid:a03f69a5) Enabled non-edge zones available for scan: 1 2025-06-29 04:06:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a18f48f5]) (logid:a03f69a5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:06:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a18f48f5]) (logid:a03f69a5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:06:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-80291bef]) (logid:1967f218) Enabled non-edge zones available for scan: 1 2025-06-29 04:06:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-80291bef]) (logid:1967f218) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:06:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ab0e93c]) (logid:c46e2110) No inactive management server node found 2025-06-29 04:06:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ab0e93c]) (logid:c46e2110) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7a43e5b]) (logid:c36e4429) No inactive management server node found 2025-06-29 04:06:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7a43e5b]) (logid:c36e4429) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:30,028 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-cb4394ae]) (logid:96acab67) HA health check task is running... 2025-06-29 04:06:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb8a9f0f]) (logid:ee1e3aee) No inactive management server node found 2025-06-29 04:06:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb8a9f0f]) (logid:ee1e3aee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-604a48f9]) (logid:1014515a) No inactive management server node found 2025-06-29 04:06:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-604a48f9]) (logid:1014515a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:34,029 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-395a6871]) (logid:2b5689d2) HA health check task is running... 2025-06-29 04:06:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfaa05b6]) (logid:13ac1cc1) No inactive management server node found 2025-06-29 04:06:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfaa05b6]) (logid:13ac1cc1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:35,285 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8464: Seq 7-8464: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:35,294 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8464 2025-06-29 04:06:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cf05a67]) (logid:3d11b8c0) No inactive management server node found 2025-06-29 04:06:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7cf05a67]) (logid:3d11b8c0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1266bfe]) (logid:e8bbddc9) No inactive management server node found 2025-06-29 04:06:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1266bfe]) (logid:e8bbddc9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:38,031 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6df563d8]) (logid:905b078c) HA health check task is running... 2025-06-29 04:06:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-984a9144]) (logid:7e502b31) No inactive management server node found 2025-06-29 04:06:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-984a9144]) (logid:7e502b31) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e465090]) (logid:3b1cd7b1) No inactive management server node found 2025-06-29 04:06:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e465090]) (logid:3b1cd7b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dae37a1]) (logid:a93948ca) No inactive management server node found 2025-06-29 04:06:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1dae37a1]) (logid:a93948ca) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:42,032 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b986e177]) (logid:92a4fd9b) HA health check task is running... 2025-06-29 04:06:43,011 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-abd38efc]) (logid:0fe103ef) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 04:06:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-0d9585f5]) (logid:7dc8b033) DbCollector is running... 2025-06-29 04:06:43,085 DEBUG [c.c.a.m.A.AgentNewConnectionsMonitorTask] (NewAgentConnectionsMonitor-1:[ctx-50080002]) (logid:8343f471) Currently 0 active new connections, of which 0 have expired - [] 2025-06-29 04:06:43,094 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-c805dbc2]) (logid:1caf1331) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:06:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-c805dbc2]) (logid:1caf1331) Snapshot scheduler is being called at [2025-06-29T02:06:43+0000]. 2025-06-29 04:06:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-c805dbc2]) (logid:1caf1331) There are [0] scheduled snapshots to be executed at [2025-06-29T02:06:43+0000]. 2025-06-29 04:06:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-aba3c114]) (logid:2915e4c6) Found 0 running routers. 2025-06-29 04:06:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a030f637]) (logid:937c0137) Found 0 routers to update status. 2025-06-29 04:06:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a030f637]) (logid:937c0137) Found 0 VPC's to update Redundant State. 2025-06-29 04:06:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a030f637]) (logid:937c0137) Found 0 networks to update RvR status. 2025-06-29 04:06:43,108 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-3481077e]) (logid:d6a4853e) Found 1 running routers. 2025-06-29 04:06:43,118 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-6037dfe4]) (logid:9ad47a54) External devices are not used. Skipping external device usage collection 2025-06-29 04:06:43,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-3481077e]) (logid:d6a4853e) Wait time setting on com.cloud.agent.api.routing.GetRouterAlertsCommand is 1800 seconds 2025-06-29 04:06:43,121 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-3481077e]) (logid:d6a4853e) Seq 2-3926857400090361994: Routed from 90520738888109 2025-06-29 04:06:43,123 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-3481077e]) (logid:d6a4853e) Seq 2-3926857400090361994: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterAlertsCommand":{"previousAlertTimeStamp":"2025-06-20 12:09:07","accessDetails":{"router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:43,128 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-a9bb533a]) (logid:bbc457da) Started resource counters recalculation periodic task. 2025-06-29 04:06:43,132 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-a9bb533a]) (logid:bbc457da) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:06:43,307 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-179b48a9]) (logid:ef8852a3) Backup backup.poll is being called at 2025-06-29T02:06:43+0000 2025-06-29 04:06:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f2c77cc]) (logid:25923ff6) No inactive management server node found 2025-06-29 04:06:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f2c77cc]) (logid:25923ff6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:43,571 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-a9bb533a]) (logid:bbc457da) Finished resource counters recalculation periodic task. 2025-06-29 04:06:43,680 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 2-3926857400090361994: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetRouterAlertsAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:43,681 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-3481077e]) (logid:d6a4853e) Seq 2-3926857400090361994: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterAlertsAnswer } } 2025-06-29 04:06:43,683 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Found 1 running routers. 2025-06-29 04:06:43,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:06:43,690 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Seq 2-3926857400090361995: Routed from 90520738888109 2025-06-29 04:06:43,690 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Seq 2-3926857400090361995: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:43,980 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-3926857400090361995: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:43,980 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Seq 2-3926857400090361995: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:06:43,980 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 04:06:43,983 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 04:06:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) ManagementServerCollector is running... 2025-06-29 04:06:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) DB properties were already loaded 2025-06-29 04:06:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:06:44,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 04:06:44,059 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Seq 2-3926857400090361996: Routed from 90520738888109 2025-06-29 04:06:44,061 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Execution of process [4066106] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:06:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Exit value of process [4066106] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:06:44,061 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Process [4066106] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:06:44,061 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) usage install: null 2025-06-29 04:06:44,061 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Sessions active: 0 2025-06-29 04:06:44,062 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Seq 2-3926857400090361996: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:06:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066109] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:06:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:06:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066114] for command [/bin/bash -c uname -r ]. 2025-06-29 04:06:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:06:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066115] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:06:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) system memory from /proc: 4887515136 2025-06-29 04:06:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:06:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066120] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:06:44,086 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) free memory from /proc: 296177664 2025-06-29 04:06:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:06:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066125] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:06:44,099 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) used memory from /proc: 1070000 2025-06-29 04:06:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:06:44,104 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066126] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:06:44,105 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:06:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066127] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:06:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:06:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066128] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:06:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:06:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066135] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:06:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:06:44,135 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Successfully executed process [4066136] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:06:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail buffersdirect.capacity, 319700 2025-06-29 04:06:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail buffersdirect.count, 93 2025-06-29 04:06:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail buffersdirect.used, 319700 2025-06-29 04:06:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail buffersmapped.capacity, 0 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail buffersmapped.count, 0 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail buffersmapped.used, 0 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail jvmuptime, 1863841 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memoryheap.committed, 500695040 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memoryheap.init, 77594624 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memoryheap.usage, 0.15841834894152534 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorynon-heap.committed, 194510848 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorynon-heap.max, -1 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorynon-heap.usage, -1.92164256E8 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorynon-heap.used, 192164256 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10778167488669688 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2108185173459526 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.Compressed-Class-Space.usage, 0.012374527752399445 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.Metaspace.usage, 0.993304797645558 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.PS-Eden-Space.usage, 0.15595549669428438 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorytotal.committed, 695205888 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorytotal.init, 85262336 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorytotal.max, 1908932607 2025-06-29 04:06:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail memorytotal.used, 494574208 2025-06-29 04:06:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail threadsdeadlocks, [] 2025-06-29 04:06:44,140 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail threadsnew.count, 0 2025-06-29 04:06:44,141 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) not storing detail threadstimed_waiting.count, 82 2025-06-29 04:06:44,142 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-bedb0817]) (logid:273794b8) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:06:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":199002192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1863765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":4.17,"totalInit":85262336,"totalUsed":493848048,"totalCommitted":695205888,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302409952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":296177664,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[4.17,1.73,0.86],"systemCyclesUsage":[66407377,53829263,4589323038],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:06:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 61, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:06:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":199002192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1863765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":4.17,"totalInit":85262336,"totalUsed":493848048,"totalCommitted":695205888,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302409952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":296177664,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[4.17,1.73,0.86],"systemCyclesUsage":[66407377,53829263,4589323038],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:06:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:06:44,148 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:06:44,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 61, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:06:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":199002192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1863765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":4.17,"totalInit":85262336,"totalUsed":493848048,"totalCommitted":695205888,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302409952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":296177664,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[4.17,1.73,0.86],"systemCyclesUsage":[66407377,53829263,4589323038],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:06:44,149 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-16:[ctx-89ce98fa]) (logid:2fff30fc) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:06:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":199002192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1863765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":4.17,"totalInit":85262336,"totalUsed":493848048,"totalCommitted":695205888,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302409952,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":296177664,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[4.17,1.73,0.86],"systemCyclesUsage":[66407377,53829263,4589323038],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:06:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9cae747]) (logid:135a1065) No inactive management server node found 2025-06-29 04:06:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9cae747]) (logid:135a1065) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:45,290 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8465: Seq 7-8465: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:45,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8465 2025-06-29 04:06:45,725 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:[]) (logid:) Seq 2-3926857400090361996: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.927f9e5b-e585-4607-99c5-8eaa6a4f0621","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:45,725 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-873a3d31]) (logid:36445702) Seq 2-3926857400090361996: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 04:06:45,727 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 04:06:45,731 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:06:45,732 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Seq 2-3926857400090361997: Routed from 90520738888109 2025-06-29 04:06:45,732 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Seq 2-3926857400090361997: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:06:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:06:46,033 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ff16e0e6]) (logid:b464b375) HA health check task is running... 2025-06-29 04:06:46,230 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 2-3926857400090361997: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "9.170770645141602", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "18.20087432861328", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "27.987003326416016", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "39.429664611816406", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162761455", "lastRunDuration": "49.861907958984375", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162761527", "lastRunDuration": "8071.785926818848", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162769650", "lastRunDuration": "61.30266189575195", "message": "Memory Usage within limits with current at 50.8732%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162769712", "lastRunDuration": "3235.4800701141357", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162773003", "lastRunDuration": "58.379411697387695", "message": "Sufficient free space is 2571.359375 MB"}, "lastRun": {"start": "2025-06-29 02:06:01.306546", "end": "2025-06-29 02:06:13.062516", "duration": "11.755970239639282"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751162401372", "lastRunDuration": "111.82308197021484", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751162401484", "lastRunDuration": "130.4488182067871", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751162401874", "lastRunDuration": "74.46980476379395", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751162402057", "lastRunDuration": "44.13628578186035", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:00:01.363182", "end": "2025-06-29 02:00:02.143261", "duration": "0.7800788879394531"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "9.170770645141602", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "18.20087432861328", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "27.987003326416016", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "39.429664611816406", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162761455", "lastRunDuration": "49.861907958984375", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162761527", "lastRunDuration": "8071.785926818848", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162769650", "lastRunDuration": "61.30266189575195", "message": "Memory Usage within limits with current at 50.8732%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162769712", "lastRunDuration": "3235.4800701141357", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162773003", "lastRunDuration": "58.379411697387695", "message": "Sufficient free space is 2571.359375 MB"}, "lastRun": {"start": "2025-06-29 02:06:01.306546", "end": "2025-06-29 02:06:13.062516", "duration": "11.755970239639282"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751162401372", "lastRunDuration": "111.82308197021484", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751162401484", "lastRunDuration": "130.4488182067871", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751162401874", "lastRunDuration": "74.46980476379395", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751162402057", "lastRunDuration": "44.13628578186035", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:00:01.363182", "end": "2025-06-29 02:00:02.143261", "duration": "0.7800788879394531"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:46,230 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Seq 2-3926857400090361997: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:06:46,230 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "9.170770645141602", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "18.20087432861328", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "27.987003326416016", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "39.429664611816406", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162761455", "lastRunDuration": "49.861907958984375", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162761527", "lastRunDuration": "8071.785926818848", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162769650", "lastRunDuration": "61.30266189575195", "message": "Memory Usage within limits with current at 50.8732%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162769712", "lastRunDuration": "3235.4800701141357", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162773003", "lastRunDuration": "58.379411697387695", "message": "Sufficient free space is 2571.359375 MB"}, "lastRun": {"start": "2025-06-29 02:06:01.306546", "end": "2025-06-29 02:06:13.062516", "duration": "11.755970239639282"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751162401372", "lastRunDuration": "111.82308197021484", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751162401484", "lastRunDuration": "130.4488182067871", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751162401874", "lastRunDuration": "74.46980476379395", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751162402057", "lastRunDuration": "44.13628578186035", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:00:01.363182", "end": "2025-06-29 02:00:02.143261", "duration": "0.7800788879394531"}}} 2025-06-29 04:06:46,263 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "9.170770645141602", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "18.20087432861328", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "27.987003326416016", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751162761308", "lastRunDuration": "39.429664611816406", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751162761455", "lastRunDuration": "49.861907958984375", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751162761527", "lastRunDuration": "8071.785926818848", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751162769650", "lastRunDuration": "61.30266189575195", "message": "Memory Usage within limits with current at 50.8732%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751162769712", "lastRunDuration": "3235.4800701141357", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751162773003", "lastRunDuration": "58.379411697387695", "message": "Sufficient free space is 2571.359375 MB"}, "lastRun": {"start": "2025-06-29 02:06:01.306546", "end": "2025-06-29 02:06:13.062516", "duration": "11.755970239639282"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751162401372", "lastRunDuration": "111.82308197021484", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751162401484", "lastRunDuration": "130.4488182067871", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751162401874", "lastRunDuration": "74.46980476379395", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751162402057", "lastRunDuration": "44.13628578186035", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:00:01.363182", "end": "2025-06-29 02:00:02.143261", "duration": "0.7800788879394531"}}} 2025-06-29 04:06:46,265 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found check types executed on VR basic, start: 2025-06-29 02:06:01.306546, end: 2025-06-29 02:06:13.062516, duration: 11.755970239639282 2025-06-29 04:06:46,275 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@52c25141- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 04:06:01 CEST 2025, details: service is running which took running duration (ms) 9.170770645141602 2025-06-29 04:06:46,286 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@77c9a828- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 04:06:01 CEST 2025, details: service is running which took running duration (ms) 18.20087432861328 2025-06-29 04:06:46,297 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@21dcb5ee- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 04:06:01 CEST 2025, details: service is running which took running duration (ms) 27.987003326416016 2025-06-29 04:06:46,309 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@199b5e4b- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 04:06:01 CEST 2025, details: service is running which took running duration (ms) 39.429664611816406 2025-06-29 04:06:46,321 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@59135c60- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 04:06:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 49.861907958984375 2025-06-29 04:06:46,334 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@63213782- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 04:06:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8071.785926818848 2025-06-29 04:06:46,344 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@27169dcd- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 04:06:09 CEST 2025, details: Memory Usage within limits with current at 50.8732% which took running duration (ms) 61.30266189575195 2025-06-29 04:06:46,355 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@643ce662- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 04:06:09 CEST 2025, details: CPU Usage within limits with current at 0.3% which took running duration (ms) 3235.4800701141357 2025-06-29 04:06:46,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0df6210a]) (logid:02dc1d15) No inactive management server node found 2025-06-29 04:06:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0df6210a]) (logid:02dc1d15) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:46,365 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@270f56d9- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 04:06:13 CEST 2025, details: Sufficient free space is 2571.359375 MB which took running duration (ms) 58.379411697387695 2025-06-29 04:06:46,365 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found check types executed on VR advanced, start: 2025-06-29 02:00:01.363182, end: 2025-06-29 02:00:02.143261, duration: 0.7800788879394531 2025-06-29 04:06:46,375 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4d78e89d- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 04:00:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 111.82308197021484 2025-06-29 04:06:46,386 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5e93fc0a- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 04:00:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 130.4488182067871 2025-06-29 04:06:46,396 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5d9f1e4c- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 04:00:01 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 74.46980476379395 2025-06-29 04:06:46,407 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-881662dc]) (logid:0b7251f6) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7b22b8- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 04:00:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 44.13628578186035 2025-06-29 04:06:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:06:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:06:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:06:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:06:47,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:06:47,070 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:06:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:06:47,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:06:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58928677]) (logid:a49cd168) No inactive management server node found 2025-06-29 04:06:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58928677]) (logid:a49cd168) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:48,045 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-1:[ctx-435cd9ca]) (logid:3864d2ba) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:06:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-cb7fcffe]) (logid:26367526) Resetting hosts suitable for reconnect 2025-06-29 04:06:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-cb7fcffe]) (logid:26367526) Completed resetting hosts suitable for reconnect 2025-06-29 04:06:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-cb7fcffe]) (logid:26367526) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:06:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-cb7fcffe]) (logid:26367526) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:06:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-cb7fcffe]) (logid:26367526) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:06:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-cb7fcffe]) (logid:26367526) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:06:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-654ca4dd]) (logid:c8d90210) VM scheduler.poll is being called at 2025-06-29T02:07:00+0000 2025-06-29 04:06:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-654ca4dd]) (logid:c8d90210) Got 0 scheduled jobs to be executed at 2025-06-29T02:07:00+0000 2025-06-29 04:06:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-654ca4dd]) (logid:c8d90210) Cleaned up 0 VM scheduled job entries 2025-06-29 04:06:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-e327846c]) (logid:79b1003c) VM Operation Thread Running 2025-06-29 04:06:48,346 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2dd0e753]) (logid:84287004) ClusterDRS.poll is being called at 2025-06-29T02:07:00+0000 2025-06-29 04:06:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2dd0e753]) (logid:84287004) Removed 0 old drs migration plans 2025-06-29 04:06:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c21ff39]) (logid:58b5288d) No inactive management server node found 2025-06-29 04:06:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c21ff39]) (logid:58b5288d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:50,034 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a5193075]) (logid:562f1ff4) HA health check task is running... 2025-06-29 04:06:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b14d1fa]) (logid:d8061a40) No inactive management server node found 2025-06-29 04:06:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b14d1fa]) (logid:d8061a40) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f841a779]) (logid:f786980c) No inactive management server node found 2025-06-29 04:06:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f841a779]) (logid:f786980c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:53,151 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-e38cccad]) (logid:41eb8300) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:06:53,163 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-e38cccad]) (logid:41eb8300) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:06:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b86e0463]) (logid:043d8061) No inactive management server node found 2025-06-29 04:06:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b86e0463]) (logid:043d8061) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:54,036 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6cb3c045]) (logid:9fa8f597) HA health check task is running... 2025-06-29 04:06:55,286 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8467: Seq 7-8467: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:06:55,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8467 2025-06-29 04:06:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adc8cf20]) (logid:172777c6) No inactive management server node found 2025-06-29 04:06:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adc8cf20]) (logid:172777c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:56,863 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddd0beb0]) (logid:4d01ca87) No inactive management server node found 2025-06-29 04:06:56,863 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddd0beb0]) (logid:4d01ca87) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:58,037 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d2bae223]) (logid:3e151d6a) HA health check task is running... 2025-06-29 04:06:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f0ef35d8]) (logid:8b532720) Enabled non-edge zones available for scan: 1 2025-06-29 04:06:58,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f0ef35d8]) (logid:8b532720) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:06:58,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f0ef35d8]) (logid:8b532720) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:06:58,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:06:58,103 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Seq 2-4725683383995203721: Routed from 90520738888109 2025-06-29 04:06:58,147 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Seq 1-4725683383995203721: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:06:58,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 04:06:58,182 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:06:58,183 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Seq 2-3926857400090361998: Routed from 90520738888109 2025-06-29 04:06:58,229 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Seq 2-3926857400090361998: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:06:58,229 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8bb795b1]) (logid:785ec9ff) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 04:06:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-19d33940]) (logid:965fdc49) Enabled non-edge zones available for scan: 1 2025-06-29 04:06:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-19d33940]) (logid:965fdc49) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:06:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d5f686a]) (logid:f4f20462) No inactive management server node found 2025-06-29 04:06:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d5f686a]) (logid:f4f20462) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:06:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7adcee2a]) (logid:32807d2b) No inactive management server node found 2025-06-29 04:06:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7adcee2a]) (logid:32807d2b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:01,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994c6200]) (logid:07ac7956) No inactive management server node found 2025-06-29 04:07:01,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-994c6200]) (logid:07ac7956) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:02,038 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ae2b323e]) (logid:a5206f72) HA health check task is running... 2025-06-29 04:07:02,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4c77a43]) (logid:ed10c3f4) No inactive management server node found 2025-06-29 04:07:02,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4c77a43]) (logid:ed10c3f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d3089b]) (logid:ffbc0e84) No inactive management server node found 2025-06-29 04:07:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d3089b]) (logid:ffbc0e84) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:05,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8468: Seq 7-8468: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:05,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8468 2025-06-29 04:07:05,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2463e4d0]) (logid:d3682df4) No inactive management server node found 2025-06-29 04:07:05,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2463e4d0]) (logid:d3682df4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:06,039 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0a57a2ec]) (logid:9c12fbdf) HA health check task is running... 2025-06-29 04:07:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91e54170]) (logid:9df6066a) No inactive management server node found 2025-06-29 04:07:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91e54170]) (logid:9df6066a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a884ae00]) (logid:77b07fed) No inactive management server node found 2025-06-29 04:07:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a884ae00]) (logid:77b07fed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:10,040 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c9a24149]) (logid:cf0f0be0) HA health check task is running... 2025-06-29 04:07:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b92d8e2b]) (logid:d51ebfb3) No inactive management server node found 2025-06-29 04:07:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b92d8e2b]) (logid:d51ebfb3) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:07:10,807 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[]) (logid:) Seq 1-4725683383995203585: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6efedcb]) (logid:cd0fe9e7) No inactive management server node found 2025-06-29 04:07:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6efedcb]) (logid:cd0fe9e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:13,003 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea82121b]) (logid:eef73b65) Checking if any host reservation can be released ... 2025-06-29 04:07:13,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea82121b]) (logid:eef73b65) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea82121b]) (logid:eef73b65) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-ea82121b]) (logid:eef73b65) Done running HostReservationReleaseChecker ... 2025-06-29 04:07:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a33ea2ed]) (logid:1146dc44) Found 0 routers to update status. 2025-06-29 04:07:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a33ea2ed]) (logid:1146dc44) Found 0 VPC's to update Redundant State. 2025-06-29 04:07:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a33ea2ed]) (logid:1146dc44) Found 0 networks to update RvR status. 2025-06-29 04:07:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Running Capacity Checker ... 2025-06-29 04:07:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) recalculating system capacity 2025-06-29 04:07:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Executing cpu/ram capacity update 2025-06-29 04:07:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:13,182 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:13,189 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:07:13,189 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:07:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:07:13,202 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:13,204 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:13,210 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:07:13,213 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:07:13,214 DEBUG [c.c.c.CapacityManagerImpl] (pool-24-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:07:13,223 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Done executing cpu/ram capacity update 2025-06-29 04:07:13,223 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Executing storage capacity update 2025-06-29 04:07:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:07:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:07:13,237 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:07:13,242 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:07:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:07:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:07:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:07:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:07:13,262 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:07:13,262 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:07:13,262 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:07:13,269 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:07:13,276 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:07:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:07:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:07:13,282 DEBUG [c.c.s.StorageManagerImpl] (pool-25-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:07:13,283 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Done executing storage capacity update 2025-06-29 04:07:13,283 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Executing capacity updates for public ip and Vlans 2025-06-29 04:07:13,302 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Done capacity updates for public ip and Vlans 2025-06-29 04:07:13,302 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Executing capacity updates for private ip 2025-06-29 04:07:13,309 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Done executing capacity updates for private ip 2025-06-29 04:07:13,309 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Done recalculating system capacity 2025-06-29 04:07:13,359 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c26e1206]) (logid:faba029a) Done running Capacity Checker ... 2025-06-29 04:07:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0470be90]) (logid:23616826) No inactive management server node found 2025-06-29 04:07:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0470be90]) (logid:23616826) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:07:14,042 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8ced5895]) (logid:30473951) HA health check task is running... 2025-06-29 04:07:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f911ecf]) (logid:34ccdfd6) No inactive management server node found 2025-06-29 04:07:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f911ecf]) (logid:34ccdfd6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:15,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8469: Seq 7-8469: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:15,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8469 2025-06-29 04:07:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77d2bd91]) (logid:b577e269) No inactive management server node found 2025-06-29 04:07:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77d2bd91]) (logid:b577e269) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bad3acc5]) (logid:7bb22db4) No inactive management server node found 2025-06-29 04:07:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bad3acc5]) (logid:7bb22db4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:17,985 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-3d133ec5]) (logid:b2b88c33) AutoScaling Monitor is running... 2025-06-29 04:07:18,043 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-52f73674]) (logid:f5cd90c4) HA health check task is running... 2025-06-29 04:07:18,046 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-f1d20fbc]) (logid:4405b964) Starting to remove all volume_stats rows older than [Sat Jun 28 16:07:18 CEST 2025]. 2025-06-29 04:07:18,046 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-f1d20fbc]) (logid:4405b964) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:07:18 CEST 2025]. 2025-06-29 04:07:18,104 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-4d06a14f]) (logid:a76f3583) Starting to remove all vm_stats rows older than [Sat Jun 28 16:07:18 CEST 2025]. 2025-06-29 04:07:18,107 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:07:18,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:07:18,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) Seq 2-4725683383995203722: Routed from 90520738888109 2025-06-29 04:07:18,128 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-4d06a14f]) (logid:a76f3583) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:07:18 CEST 2025]. 2025-06-29 04:07:18,154 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) Seq 1-4725683383995203722: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:07:18,156 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:07:18,157 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) Seq 2-3926857400090361999: Routed from 90520738888109 2025-06-29 04:07:18,201 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-60de81b7]) (logid:5d3ad473) Seq 2-3926857400090361999: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:07:18,355 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:07:18,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:07:18,360 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) Seq 2-4725683383995203723: Routed from 90520738888109 2025-06-29 04:07:18,416 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) Seq 1-4725683383995203723: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:07:18,453 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:07:18,454 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) Seq 2-3926857400090362000: Routed from 90520738888109 2025-06-29 04:07:18,465 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) StorageCollector is running... 2025-06-29 04:07:18,475 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:07:18,475 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:07:18,475 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:07:18,476 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-3938116399158788134: Routed from 90520738888109 2025-06-29 04:07:18,533 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 4-3938116399158788134: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:07:18,537 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:07:18,537 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:07:18,537 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:07:18,539 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-3926857400090362001: Routed from 90520738888109 2025-06-29 04:07:18,546 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2a1f2c83]) (logid:67205dad) Seq 2-3926857400090362000: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:07:18,553 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-3926857400090362001: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:07:18,565 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:07:18,566 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:07:18,566 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:07:18,568 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-4725683383995203724: Routed from 90520738888109 2025-06-29 04:07:18,615 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 1-4725683383995203724: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:07:18,626 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:07:18,626 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:07:18,626 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:07:18,627 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-4725683383995203725: Routed from 90520738888109 2025-06-29 04:07:18,711 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 1-4725683383995203725: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:07:18,713 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:07:18,713 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:07:18,713 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:07:18,714 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-3926857400090362002: Routed from 90520738888109 2025-06-29 04:07:18,760 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0977def9]) (logid:663a6bab) Seq 2-3926857400090362002: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:07:19,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-558c9715]) (logid:31817573) No inactive management server node found 2025-06-29 04:07:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-558c9715]) (logid:31817573) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ebab999]) (logid:0be56169) No inactive management server node found 2025-06-29 04:07:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ebab999]) (logid:0be56169) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:22,045 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b5451a21]) (logid:4d86899f) HA health check task is running... 2025-06-29 04:07:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4742d7a8]) (logid:3c8680c8) No inactive management server node found 2025-06-29 04:07:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4742d7a8]) (logid:3c8680c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c0de661]) (logid:867c6071) No inactive management server node found 2025-06-29 04:07:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c0de661]) (logid:867c6071) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:25,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8470: Seq 7-8470: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:25,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8470 2025-06-29 04:07:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3506845]) (logid:233d7119) No inactive management server node found 2025-06-29 04:07:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3506845]) (logid:233d7119) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:26,046 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bb5c4d14]) (logid:7bce3afd) HA health check task is running... 2025-06-29 04:07:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2217b56]) (logid:5f7079b4) No inactive management server node found 2025-06-29 04:07:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2217b56]) (logid:5f7079b4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-959459c9]) (logid:9514bad0) Enabled non-edge zones available for scan: 1 2025-06-29 04:07:28,099 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-959459c9]) (logid:9514bad0) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:07:28,101 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-959459c9]) (logid:9514bad0) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:07:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ef71074b]) (logid:1a5ad10c) Enabled non-edge zones available for scan: 1 2025-06-29 04:07:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ef71074b]) (logid:1a5ad10c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:07:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a4448df]) (logid:4dc615d7) No inactive management server node found 2025-06-29 04:07:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a4448df]) (logid:4dc615d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86b18f84]) (logid:4ae570cb) No inactive management server node found 2025-06-29 04:07:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86b18f84]) (logid:4ae570cb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:30,047 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4bc0b0e9]) (logid:0f1f8c10) HA health check task is running... 2025-06-29 04:07:31,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e8c691f6]) (logid:86e7dfe5) No inactive management server node found 2025-06-29 04:07:31,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e8c691f6]) (logid:86e7dfe5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7af66d5a]) (logid:e347f559) No inactive management server node found 2025-06-29 04:07:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7af66d5a]) (logid:e347f559) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:34,048 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1f2ada5f]) (logid:a7e5cd2f) HA health check task is running... 2025-06-29 04:07:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24e8cb63]) (logid:c641a2b2) No inactive management server node found 2025-06-29 04:07:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24e8cb63]) (logid:c641a2b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:35,288 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8471: Seq 7-8471: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:35,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8471 2025-06-29 04:07:35,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d8aa28]) (logid:16a86dbf) No inactive management server node found 2025-06-29 04:07:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23d8aa28]) (logid:16a86dbf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87ea4cf4]) (logid:64fee3f5) No inactive management server node found 2025-06-29 04:07:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87ea4cf4]) (logid:64fee3f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:38,050 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2eb7324a]) (logid:4c85ec79) HA health check task is running... 2025-06-29 04:07:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fa414fe]) (logid:d55f2df0) No inactive management server node found 2025-06-29 04:07:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fa414fe]) (logid:d55f2df0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:40,288 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8472: Seq 7-8472: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:40,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8472 2025-06-29 04:07:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad73e86c]) (logid:0c079c79) No inactive management server node found 2025-06-29 04:07:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad73e86c]) (logid:0c079c79) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-837e8bd6]) (logid:7a3be579) No inactive management server node found 2025-06-29 04:07:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-837e8bd6]) (logid:7a3be579) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:42,051 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-08a7a899]) (logid:47eb114a) HA health check task is running... 2025-06-29 04:07:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-5f124bd4]) (logid:c623a205) DbCollector is running... 2025-06-29 04:07:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-26216f89]) (logid:8c0a23f0) Found 0 routers to update status. 2025-06-29 04:07:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-26216f89]) (logid:8c0a23f0) Found 0 VPC's to update Redundant State. 2025-06-29 04:07:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-26216f89]) (logid:8c0a23f0) Found 0 networks to update RvR status. 2025-06-29 04:07:43,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59115ca7]) (logid:d7d4d4c2) No inactive management server node found 2025-06-29 04:07:43,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59115ca7]) (logid:d7d4d4c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) ManagementServerCollector is running... 2025-06-29 04:07:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) DB properties were already loaded 2025-06-29 04:07:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:07:44,058 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Execution of process [4066144] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:07:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Exit value of process [4066144] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:07:44,058 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Process [4066144] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:07:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) usage install: null 2025-06-29 04:07:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Sessions active: 0 2025-06-29 04:07:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:07:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066147] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:07:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:07:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066152] for command [/bin/bash -c uname -r ]. 2025-06-29 04:07:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:07:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066153] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:07:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) system memory from /proc: 4887515136 2025-06-29 04:07:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:07:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066158] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:07:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) free memory from /proc: 295866368 2025-06-29 04:07:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:07:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066163] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:07:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) used memory from /proc: 1070000 2025-06-29 04:07:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:07:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066164] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:07:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:07:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066165] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:07:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:07:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066166] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:07:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:07:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066173] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:07:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:07:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Successfully executed process [4066174] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail buffersdirect.capacity, 321654 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail buffersdirect.count, 93 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail buffersdirect.used, 321654 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail buffersmapped.capacity, 0 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail buffersmapped.count, 0 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail buffersmapped.used, 0 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:07:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail jvmuptime, 1923831 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memoryheap.committed, 500695040 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memoryheap.init, 77594624 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memoryheap.usage, 0.17943261829387747 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorynon-heap.committed, 194576384 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorynon-heap.max, -1 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorynon-heap.usage, -1.92341496E8 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorynon-heap.used, 192341496 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10815657491335644 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2115287682873996 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.Compressed-Class-Space.usage, 0.012379668653011322 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.Metaspace.usage, 0.9931122401975235 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.PS-Eden-Space.usage, 0.21846168201982635 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorytotal.committed, 695271424 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorytotal.init, 85262336 2025-06-29 04:07:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorytotal.max, 1908932607 2025-06-29 04:07:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail memorytotal.used, 535092848 2025-06-29 04:07:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail threadsdeadlocks, [] 2025-06-29 04:07:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail threadsnew.count, 0 2025-06-29 04:07:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) not storing detail threadstimed_waiting.count, 82 2025-06-29 04:07:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-b13f7bc3]) (logid:93578cfa) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:07:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":158890952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1923761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.53,"totalInit":85262336,"totalUsed":534144304,"totalCommitted":695271424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":342524776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":295866368,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.53,1.42,0.8],"systemCyclesUsage":[66407787,53829616,4589346017],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:07:44,133 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 63, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:07:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":158890952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1923761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.53,"totalInit":85262336,"totalUsed":534144304,"totalCommitted":695271424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":342524776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":295866368,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.53,1.42,0.8],"systemCyclesUsage":[66407787,53829616,4589346017],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:07:44,133 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:07:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-16:[ctx-fa4da9cf]) (logid:9bab2703) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:07:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":158890952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1923761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.53,"totalInit":85262336,"totalUsed":534144304,"totalCommitted":695271424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":342524776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":295866368,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.53,1.42,0.8],"systemCyclesUsage":[66407787,53829616,4589346017],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:07:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:07:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 63, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:07:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":158890952,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1923761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":1.53,"totalInit":85262336,"totalUsed":534144304,"totalCommitted":695271424,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":342524776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":295866368,"systemMemoryUsed":1070000,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[1.53,1.42,0.8],"systemCyclesUsage":[66407787,53829616,4589346017],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:07:44,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ae960db]) (logid:073fca04) No inactive management server node found 2025-06-29 04:07:44,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ae960db]) (logid:073fca04) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:07:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:07:46,052 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-99a5e29a]) (logid:24eb87d7) HA health check task is running... 2025-06-29 04:07:46,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-539bb5ee]) (logid:c7bea093) No inactive management server node found 2025-06-29 04:07:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-539bb5ee]) (logid:c7bea093) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:07:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:07:47,088 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:07:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:07:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed4594ab]) (logid:c791c182) No inactive management server node found 2025-06-29 04:07:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed4594ab]) (logid:c791c182) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-888f0dc6]) (logid:b578a571) VM scheduler.poll is being called at 2025-06-29T02:08:00+0000 2025-06-29 04:07:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-888f0dc6]) (logid:b578a571) Got 0 scheduled jobs to be executed at 2025-06-29T02:08:00+0000 2025-06-29 04:07:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-888f0dc6]) (logid:b578a571) Cleaned up 0 VM scheduled job entries 2025-06-29 04:07:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-8f401af1]) (logid:c38c1b61) VM Operation Thread Running 2025-06-29 04:07:48,346 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-c6f1b0d1]) (logid:c164036f) ClusterDRS.poll is being called at 2025-06-29T02:08:00+0000 2025-06-29 04:07:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-c6f1b0d1]) (logid:c164036f) Removed 0 old drs migration plans 2025-06-29 04:07:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ce0fca9]) (logid:2bc7f36c) No inactive management server node found 2025-06-29 04:07:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ce0fca9]) (logid:2bc7f36c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:50,053 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-31279a7c]) (logid:eeaef52e) HA health check task is running... 2025-06-29 04:07:50,290 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8474: Seq 7-8474: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:07:50,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8474 2025-06-29 04:07:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-640daa32]) (logid:f85216d5) No inactive management server node found 2025-06-29 04:07:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-640daa32]) (logid:f85216d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b8575e1]) (logid:5eb02dea) No inactive management server node found 2025-06-29 04:07:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b8575e1]) (logid:5eb02dea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b50653c]) (logid:e8c9a6b5) No inactive management server node found 2025-06-29 04:07:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b50653c]) (logid:e8c9a6b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:54,054 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c5bc8ada]) (logid:51700be2) HA health check task is running... 2025-06-29 04:07:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db85c3b0]) (logid:f8e2e899) No inactive management server node found 2025-06-29 04:07:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db85c3b0]) (logid:f8e2e899) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f11eaf2]) (logid:fbb7f052) No inactive management server node found 2025-06-29 04:07:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f11eaf2]) (logid:fbb7f052) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:58,055 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7fc8058c]) (logid:de5c13de) HA health check task is running... 2025-06-29 04:07:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-82a6ec54]) (logid:6ef4ea56) Enabled non-edge zones available for scan: 1 2025-06-29 04:07:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-82a6ec54]) (logid:6ef4ea56) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:07:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-82a6ec54]) (logid:6ef4ea56) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:07:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f5ec4da1]) (logid:b19b9a4a) Enabled non-edge zones available for scan: 1 2025-06-29 04:07:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f5ec4da1]) (logid:b19b9a4a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:07:58,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a895fb8]) (logid:3a1a597a) No inactive management server node found 2025-06-29 04:07:58,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a895fb8]) (logid:3a1a597a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:07:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7998d24]) (logid:059e1186) No inactive management server node found 2025-06-29 04:07:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7998d24]) (logid:059e1186) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:00,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8475: Seq 7-8475: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:08:00,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8475 2025-06-29 04:08:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b1c3385]) (logid:c0b8878c) No inactive management server node found 2025-06-29 04:08:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b1c3385]) (logid:c0b8878c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:02,056 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5e7036a5]) (logid:9c146892) HA health check task is running... 2025-06-29 04:08:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af712dd2]) (logid:3b925f7a) No inactive management server node found 2025-06-29 04:08:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af712dd2]) (logid:3b925f7a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0d5186a]) (logid:4b5ec256) No inactive management server node found 2025-06-29 04:08:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0d5186a]) (logid:4b5ec256) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45852ae8]) (logid:10a3233c) No inactive management server node found 2025-06-29 04:08:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45852ae8]) (logid:10a3233c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:06,057 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9cb44b68]) (logid:b98a7fc6) HA health check task is running... 2025-06-29 04:08:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-987eff57]) (logid:12da5ae9) No inactive management server node found 2025-06-29 04:08:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-987eff57]) (logid:12da5ae9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5f74679]) (logid:b238a0d9) No inactive management server node found 2025-06-29 04:08:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5f74679]) (logid:b238a0d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:10,058 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d1cc5dc1]) (logid:a58c9a66) HA health check task is running... 2025-06-29 04:08:10,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8476: Seq 7-8476: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:08:10,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8476 2025-06-29 04:08:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36f62a9e]) (logid:974e612d) No inactive management server node found 2025-06-29 04:08:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36f62a9e]) (logid:974e612d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09ace14f]) (logid:54e7148a) No inactive management server node found 2025-06-29 04:08:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09ace14f]) (logid:54e7148a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7449d40e]) (logid:06094d92) Found 0 routers to update status. 2025-06-29 04:08:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7449d40e]) (logid:06094d92) Found 0 VPC's to update Redundant State. 2025-06-29 04:08:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7449d40e]) (logid:06094d92) Found 0 networks to update RvR status. 2025-06-29 04:08:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c905f515]) (logid:a4813b81) No inactive management server node found 2025-06-29 04:08:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c905f515]) (logid:a4813b81) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:14,060 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1f3c88be]) (logid:c1f0d0aa) HA health check task is running... 2025-06-29 04:08:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25dd87a0]) (logid:cecb88c5) No inactive management server node found 2025-06-29 04:08:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25dd87a0]) (logid:cecb88c5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-842a8654]) (logid:82a555f3) No inactive management server node found 2025-06-29 04:08:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-842a8654]) (logid:82a555f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd444e66]) (logid:f3f97616) No inactive management server node found 2025-06-29 04:08:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd444e66]) (logid:f3f97616) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:17,988 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-ead6470d]) (logid:b0a5e793) AutoScaling Monitor is running... 2025-06-29 04:08:18,048 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-78b7fdda]) (logid:665b2089) Starting to remove all volume_stats rows older than [Sat Jun 28 16:08:18 CEST 2025]. 2025-06-29 04:08:18,049 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-78b7fdda]) (logid:665b2089) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:08:18 CEST 2025]. 2025-06-29 04:08:18,061 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cdcdf5d0]) (logid:2f737ed3) HA health check task is running... 2025-06-29 04:08:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-db397449]) (logid:6d8b85b7) Resetting hosts suitable for reconnect 2025-06-29 04:08:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-db397449]) (logid:6d8b85b7) Completed resetting hosts suitable for reconnect 2025-06-29 04:08:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-db397449]) (logid:6d8b85b7) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:08:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-db397449]) (logid:6d8b85b7) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:08:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-db397449]) (logid:6d8b85b7) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:08:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-db397449]) (logid:6d8b85b7) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:08:18,130 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-92f6cc2d]) (logid:adbe0c0f) Starting to remove all vm_stats rows older than [Sat Jun 28 16:08:18 CEST 2025]. 2025-06-29 04:08:18,154 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-92f6cc2d]) (logid:adbe0c0f) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:08:18 CEST 2025]. 2025-06-29 04:08:18,205 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:08:18,207 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:08:18,208 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) Seq 2-4725683383995203726: Routed from 90520738888109 2025-06-29 04:08:18,252 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) Seq 1-4725683383995203726: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:08:18,253 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:08:18,254 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) Seq 2-3926857400090362003: Routed from 90520738888109 2025-06-29 04:08:18,303 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bdf5cd44]) (logid:e97c05f2) Seq 2-3926857400090362003: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:08:18,601 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:08:18,605 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:08:18,605 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) Seq 2-4725683383995203727: Routed from 90520738888109 2025-06-29 04:08:18,659 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) Seq 1-4725683383995203727: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:08:18,693 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:08:18,694 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) Seq 2-3926857400090362004: Routed from 90520738888109 2025-06-29 04:08:18,760 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) StorageCollector is running... 2025-06-29 04:08:18,767 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:08:18,767 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:08:18,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:08:18,768 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3938116399158788135: Routed from 90520738888109 2025-06-29 04:08:18,781 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-d82f693c]) (logid:1f73b295) Seq 2-3926857400090362004: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:08:18,822 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 4-3938116399158788135: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:08:18,827 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:08:18,827 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:08:18,827 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:08:18,829 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3926857400090362005: Routed from 90520738888109 2025-06-29 04:08:18,840 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3926857400090362005: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:08:18,859 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:08:18,860 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:08:18,860 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:08:18,861 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3926857400090362006: Routed from 90520738888109 2025-06-29 04:08:18,954 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3926857400090362006: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:08:18,965 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:08:18,965 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:08:18,965 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:08:18,966 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-4725683383995203728: Routed from 90520738888109 2025-06-29 04:08:19,014 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 1-4725683383995203728: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:08:19,016 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:08:19,016 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:08:19,016 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:08:19,018 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3926857400090362007: Routed from 90520738888109 2025-06-29 04:08:19,063 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d54fdf3b]) (logid:4e33efae) Seq 2-3926857400090362007: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:08:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db643350]) (logid:a151cbe6) No inactive management server node found 2025-06-29 04:08:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db643350]) (logid:a151cbe6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:20,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8477: Seq 7-8477: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:08:20,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8477 2025-06-29 04:08:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0130d33]) (logid:71005cc4) No inactive management server node found 2025-06-29 04:08:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0130d33]) (logid:71005cc4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:22,062 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d908e5b8]) (logid:7b543a3c) HA health check task is running... 2025-06-29 04:08:22,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f07961c4]) (logid:e6456b3d) No inactive management server node found 2025-06-29 04:08:22,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f07961c4]) (logid:e6456b3d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b76b1ff]) (logid:0b01e8a5) No inactive management server node found 2025-06-29 04:08:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b76b1ff]) (logid:0b01e8a5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-074925e0]) (logid:bce3cf73) No inactive management server node found 2025-06-29 04:08:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-074925e0]) (logid:bce3cf73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:26,063 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-255572ce]) (logid:05e58e88) HA health check task is running... 2025-06-29 04:08:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c2640ae]) (logid:6a0f9557) No inactive management server node found 2025-06-29 04:08:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c2640ae]) (logid:6a0f9557) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-79d9759e]) (logid:3d88bb2f) Enabled non-edge zones available for scan: 1 2025-06-29 04:08:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-79d9759e]) (logid:3d88bb2f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:08:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-79d9759e]) (logid:3d88bb2f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:08:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7fe2ca9c]) (logid:a08d3603) Enabled non-edge zones available for scan: 1 2025-06-29 04:08:28,285 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7fe2ca9c]) (logid:a08d3603) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:08:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fda5ef8]) (logid:9d6535ea) No inactive management server node found 2025-06-29 04:08:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fda5ef8]) (logid:9d6535ea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-946992c5]) (logid:4a81681d) No inactive management server node found 2025-06-29 04:08:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-946992c5]) (logid:4a81681d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:30,064 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bf78623b]) (logid:ffd2955f) HA health check task is running... 2025-06-29 04:08:30,290 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8478: Seq 7-8478: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:08:30,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8478 2025-06-29 04:08:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc335108]) (logid:84dd7e39) No inactive management server node found 2025-06-29 04:08:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc335108]) (logid:84dd7e39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d408a2a3]) (logid:b36e6f67) No inactive management server node found 2025-06-29 04:08:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d408a2a3]) (logid:b36e6f67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:34,065 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-66f4005c]) (logid:6223daf6) HA health check task is running... 2025-06-29 04:08:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24216e26]) (logid:82fbad0c) No inactive management server node found 2025-06-29 04:08:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24216e26]) (logid:82fbad0c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dbf7a87]) (logid:59b8987c) No inactive management server node found 2025-06-29 04:08:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dbf7a87]) (logid:59b8987c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9845b11]) (logid:57128807) No inactive management server node found 2025-06-29 04:08:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9845b11]) (logid:57128807) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:38,066 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-766b4783]) (logid:bbef1812) HA health check task is running... 2025-06-29 04:08:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bec7270]) (logid:7c1767cb) No inactive management server node found 2025-06-29 04:08:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bec7270]) (logid:7c1767cb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:40,291 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8479: Seq 7-8479: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:08:40,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8479 2025-06-29 04:08:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69dad56a]) (logid:bddb28f1) No inactive management server node found 2025-06-29 04:08:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69dad56a]) (logid:bddb28f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbc2fa4e]) (logid:dbea77d9) No inactive management server node found 2025-06-29 04:08:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbc2fa4e]) (logid:dbea77d9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:42,067 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-664ee834]) (logid:26683986) HA health check task is running... 2025-06-29 04:08:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-c0d6674e]) (logid:6b386fb9) DbCollector is running... 2025-06-29 04:08:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-417a9c03]) (logid:73398704) Found 0 routers to update status. 2025-06-29 04:08:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-417a9c03]) (logid:73398704) Found 0 VPC's to update Redundant State. 2025-06-29 04:08:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-417a9c03]) (logid:73398704) Found 0 networks to update RvR status. 2025-06-29 04:08:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b6e5fa8]) (logid:daeffa2d) No inactive management server node found 2025-06-29 04:08:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b6e5fa8]) (logid:daeffa2d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:08:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) ManagementServerCollector is running... 2025-06-29 04:08:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) DB properties were already loaded 2025-06-29 04:08:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:08:44,056 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Execution of process [4066187] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:08:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Exit value of process [4066187] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:08:44,057 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Process [4066187] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:08:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) usage install: null 2025-06-29 04:08:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Sessions active: 0 2025-06-29 04:08:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:08:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066190] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:08:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:08:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066195] for command [/bin/bash -c uname -r ]. 2025-06-29 04:08:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:08:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066196] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:08:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) system memory from /proc: 4887515136 2025-06-29 04:08:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:08:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066201] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:08:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) free memory from /proc: 300101632 2025-06-29 04:08:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:08:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066206] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:08:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) used memory from /proc: 1070256 2025-06-29 04:08:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:08:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066207] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:08:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:08:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066208] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:08:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:08:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066209] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:08:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:08:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066216] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:08:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:08:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Successfully executed process [4066217] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:08:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail buffersdirect.capacity, 321654 2025-06-29 04:08:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail buffersdirect.count, 93 2025-06-29 04:08:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail buffersdirect.used, 321654 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail buffersmapped.capacity, 0 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail buffersmapped.count, 0 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail buffersmapped.used, 0 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail jvmuptime, 1983835 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memoryheap.committed, 500695040 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memoryheap.init, 77594624 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memoryheap.usage, 0.19151446125855062 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorynon-heap.committed, 194707456 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorynon-heap.max, -1 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorynon-heap.usage, -1.92523552E8 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorynon-heap.used, 192523552 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1087439182884564 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2121536224880861 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.Compressed-Class-Space.usage, 0.012379668653011322 2025-06-29 04:08:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.Metaspace.usage, 0.9933502197265625 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.PS-Eden-Space.usage, 0.2538457850340805 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorytotal.committed, 695402496 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorytotal.init, 85262336 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorytotal.max, 1908932607 2025-06-29 04:08:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail memorytotal.used, 558111752 2025-06-29 04:08:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail threadsdeadlocks, [] 2025-06-29 04:08:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail threadsnew.count, 0 2025-06-29 04:08:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) not storing detail threadstimed_waiting.count, 82 2025-06-29 04:08:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-3e3c34d6]) (logid:7d8838cd) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:08:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":135870656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1983761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":557347936,"totalCommitted":695402496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":365588200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":300101632,"systemMemoryUsed":1070256,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,1.17,0.75],"systemCyclesUsage":[66408142,53829915,4589369123],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:08:44,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 65, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:08:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":135870656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1983761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":557347936,"totalCommitted":695402496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":365588200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":300101632,"systemMemoryUsed":1070256,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,1.17,0.75],"systemCyclesUsage":[66408142,53829915,4589369123],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:08:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:08:44,145 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-17:[ctx-923800ec]) (logid:86e445a8) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:08:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":135870656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1983761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":557347936,"totalCommitted":695402496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":365588200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":300101632,"systemMemoryUsed":1070256,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,1.17,0.75],"systemCyclesUsage":[66408142,53829915,4589369123],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:08:44,145 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:08:44,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 65, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:08:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":135870656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1983761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":557347936,"totalCommitted":695402496,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":365588200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":300101632,"systemMemoryUsed":1070256,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,1.17,0.75],"systemCyclesUsage":[66408142,53829915,4589369123],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:08:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e38fadd]) (logid:19b333c5) No inactive management server node found 2025-06-29 04:08:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e38fadd]) (logid:19b333c5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:08:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:08:46,068 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-363af8a8]) (logid:076865ba) HA health check task is running... 2025-06-29 04:08:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c8a2c3b]) (logid:7bdf1123) No inactive management server node found 2025-06-29 04:08:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c8a2c3b]) (logid:7bdf1123) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:08:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:08:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:08:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:08:47,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:08:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:08:47,101 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:08:47,103 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:08:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86a3d519]) (logid:e39581a9) No inactive management server node found 2025-06-29 04:08:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86a3d519]) (logid:e39581a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c7f22d36]) (logid:b1f076b8) VM scheduler.poll is being called at 2025-06-29T02:09:00+0000 2025-06-29 04:08:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c7f22d36]) (logid:b1f076b8) Got 0 scheduled jobs to be executed at 2025-06-29T02:09:00+0000 2025-06-29 04:08:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c7f22d36]) (logid:b1f076b8) Cleaned up 0 VM scheduled job entries 2025-06-29 04:08:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-114eab25]) (logid:4745584e) VM Operation Thread Running 2025-06-29 04:08:48,346 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5ed1410c]) (logid:59eef842) ClusterDRS.poll is being called at 2025-06-29T02:09:00+0000 2025-06-29 04:08:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5ed1410c]) (logid:59eef842) Removed 0 old drs migration plans 2025-06-29 04:08:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db0919de]) (logid:9f03398a) No inactive management server node found 2025-06-29 04:08:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db0919de]) (logid:9f03398a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:50,069 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1d9198d7]) (logid:0919b124) HA health check task is running... 2025-06-29 04:08:50,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8481: Seq 7-8481: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:08:50,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8481 2025-06-29 04:08:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6debe287]) (logid:2bbb99ed) No inactive management server node found 2025-06-29 04:08:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6debe287]) (logid:2bbb99ed) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6a1cee9]) (logid:27640844) No inactive management server node found 2025-06-29 04:08:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6a1cee9]) (logid:27640844) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f248d3f]) (logid:4de4e2d6) No inactive management server node found 2025-06-29 04:08:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f248d3f]) (logid:4de4e2d6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:54,070 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3f212408]) (logid:e1c0492b) HA health check task is running... 2025-06-29 04:08:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9b0f06c]) (logid:3de28429) No inactive management server node found 2025-06-29 04:08:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9b0f06c]) (logid:3de28429) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-059d8581]) (logid:30818383) No inactive management server node found 2025-06-29 04:08:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-059d8581]) (logid:30818383) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:58,071 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0eb18c33]) (logid:1c741e54) HA health check task is running... 2025-06-29 04:08:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c6fa29f5]) (logid:45b00b52) Enabled non-edge zones available for scan: 1 2025-06-29 04:08:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c6fa29f5]) (logid:45b00b52) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:08:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c6fa29f5]) (logid:45b00b52) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:08:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c62a3f09]) (logid:d85d13fb) Enabled non-edge zones available for scan: 1 2025-06-29 04:08:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c62a3f09]) (logid:d85d13fb) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:08:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3b5a968]) (logid:60298ea9) No inactive management server node found 2025-06-29 04:08:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3b5a968]) (logid:60298ea9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:08:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a2f5ce2]) (logid:3e572d94) No inactive management server node found 2025-06-29 04:08:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a2f5ce2]) (logid:3e572d94) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:00,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8482: Seq 7-8482: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:09:00,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8482 2025-06-29 04:09:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8350b849]) (logid:fb5c10cc) No inactive management server node found 2025-06-29 04:09:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8350b849]) (logid:fb5c10cc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:02,072 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-724a3103]) (logid:99766d1f) HA health check task is running... 2025-06-29 04:09:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57b063a7]) (logid:9b7c9552) No inactive management server node found 2025-06-29 04:09:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57b063a7]) (logid:9b7c9552) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bce5f46]) (logid:282c6090) No inactive management server node found 2025-06-29 04:09:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bce5f46]) (logid:282c6090) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa6b95d1]) (logid:32fde715) No inactive management server node found 2025-06-29 04:09:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa6b95d1]) (logid:32fde715) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:06,073 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-aae411d1]) (logid:e5b0c21f) HA health check task is running... 2025-06-29 04:09:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c55cf5ce]) (logid:3a65e732) No inactive management server node found 2025-06-29 04:09:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c55cf5ce]) (logid:3a65e732) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ed5d534]) (logid:4cce011a) No inactive management server node found 2025-06-29 04:09:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ed5d534]) (logid:4cce011a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:10,074 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-282ab8d9]) (logid:33d94a9d) HA health check task is running... 2025-06-29 04:09:10,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8483: Seq 7-8483: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:09:10,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8483 2025-06-29 04:09:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3afa8c4f]) (logid:3b296ab1) No inactive management server node found 2025-06-29 04:09:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3afa8c4f]) (logid:3b296ab1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b69a933]) (logid:b496b621) No inactive management server node found 2025-06-29 04:09:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b69a933]) (logid:b496b621) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-251341f0]) (logid:b1565878) Found 0 routers to update status. 2025-06-29 04:09:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-251341f0]) (logid:b1565878) Found 0 VPC's to update Redundant State. 2025-06-29 04:09:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-251341f0]) (logid:b1565878) Found 0 networks to update RvR status. 2025-06-29 04:09:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bda1fd6d]) (logid:ed531c20) No inactive management server node found 2025-06-29 04:09:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bda1fd6d]) (logid:ed531c20) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:14,075 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-fc31b0a4]) (logid:d608d5dd) HA health check task is running... 2025-06-29 04:09:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e98ed96e]) (logid:e650562e) No inactive management server node found 2025-06-29 04:09:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e98ed96e]) (logid:e650562e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ed656bb]) (logid:583a6c83) No inactive management server node found 2025-06-29 04:09:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ed656bb]) (logid:583a6c83) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fafa314]) (logid:d6844bcb) No inactive management server node found 2025-06-29 04:09:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fafa314]) (logid:d6844bcb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:17,991 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-ad4f7865]) (logid:4b59d1f5) AutoScaling Monitor is running... 2025-06-29 04:09:18,051 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-5b90f9f9]) (logid:782fad4d) Starting to remove all volume_stats rows older than [Sat Jun 28 16:09:18 CEST 2025]. 2025-06-29 04:09:18,052 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-5b90f9f9]) (logid:782fad4d) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:09:18 CEST 2025]. 2025-06-29 04:09:18,076 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9cc8efb1]) (logid:9ed7d95a) HA health check task is running... 2025-06-29 04:09:18,156 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-374091a3]) (logid:5eed7070) Starting to remove all vm_stats rows older than [Sat Jun 28 16:09:18 CEST 2025]. 2025-06-29 04:09:18,181 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-374091a3]) (logid:5eed7070) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:09:18 CEST 2025]. 2025-06-29 04:09:18,306 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:09:18,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:09:18,310 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) Seq 2-4725683383995203729: Routed from 90520738888109 2025-06-29 04:09:18,354 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) Seq 1-4725683383995203729: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:09:18,355 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:09:18,355 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) Seq 2-3926857400090362008: Routed from 90520738888109 2025-06-29 04:09:18,400 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-45a5311f]) (logid:a6790e39) Seq 2-3926857400090362008: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:09:18,836 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:09:18,839 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:09:18,840 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) Seq 2-4725683383995203730: Routed from 90520738888109 2025-06-29 04:09:18,893 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) Seq 1-4725683383995203730: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:09:18,929 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:09:18,929 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) Seq 2-3926857400090362009: Routed from 90520738888109 2025-06-29 04:09:19,022 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-da65a0ab]) (logid:3e112f04) Seq 2-3926857400090362009: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:09:19,063 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) StorageCollector is running... 2025-06-29 04:09:19,069 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:09:19,069 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:09:19,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:09:19,070 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3938116399158788136: Routed from 90520738888109 2025-06-29 04:09:19,126 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 4-3938116399158788136: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:09:19,130 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:09:19,130 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:09:19,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:09:19,131 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3926857400090362010: Routed from 90520738888109 2025-06-29 04:09:19,143 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3926857400090362010: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:09:19,153 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:09:19,153 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:09:19,153 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:09:19,155 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3926857400090362011: Routed from 90520738888109 2025-06-29 04:09:19,248 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3926857400090362011: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:09:19,258 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:09:19,258 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:09:19,258 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:09:19,260 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-4725683383995203731: Routed from 90520738888109 2025-06-29 04:09:19,303 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 1-4725683383995203731: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:09:19,305 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:09:19,305 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:09:19,305 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:09:19,306 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3926857400090362012: Routed from 90520738888109 2025-06-29 04:09:19,351 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-147f6328]) (logid:35cdfeac) Seq 2-3926857400090362012: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:09:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3da59256]) (logid:ae0511b4) No inactive management server node found 2025-06-29 04:09:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3da59256]) (logid:ae0511b4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:20,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8484: Seq 7-8484: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:09:20,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8484 2025-06-29 04:09:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0691a10]) (logid:cc64da82) No inactive management server node found 2025-06-29 04:09:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0691a10]) (logid:cc64da82) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:22,076 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-dda3edd0]) (logid:b1f9d153) HA health check task is running... 2025-06-29 04:09:22,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3969292d]) (logid:20287062) No inactive management server node found 2025-06-29 04:09:22,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3969292d]) (logid:20287062) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-116df70f]) (logid:86fcb55a) No inactive management server node found 2025-06-29 04:09:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-116df70f]) (logid:86fcb55a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ded72d56]) (logid:7fab8356) No inactive management server node found 2025-06-29 04:09:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ded72d56]) (logid:7fab8356) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:26,078 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1c957986]) (logid:3c8bb7ce) HA health check task is running... 2025-06-29 04:09:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76885de3]) (logid:fcc77e11) No inactive management server node found 2025-06-29 04:09:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76885de3]) (logid:fcc77e11) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c49df0c]) (logid:44e7a5a9) Enabled non-edge zones available for scan: 1 2025-06-29 04:09:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0c49df0c]) (logid:44e7a5a9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:09:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c49df0c]) (logid:44e7a5a9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:09:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7ac0e23e]) (logid:1e01c5fe) Enabled non-edge zones available for scan: 1 2025-06-29 04:09:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7ac0e23e]) (logid:1e01c5fe) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:09:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d92ef12]) (logid:8f4a1ead) No inactive management server node found 2025-06-29 04:09:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d92ef12]) (logid:8f4a1ead) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72d5676c]) (logid:03513bce) No inactive management server node found 2025-06-29 04:09:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72d5676c]) (logid:03513bce) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:30,078 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d1529538]) (logid:12362f73) HA health check task is running... 2025-06-29 04:09:30,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8485: Seq 7-8485: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:09:30,304 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8485 2025-06-29 04:09:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f888a17]) (logid:99671519) No inactive management server node found 2025-06-29 04:09:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f888a17]) (logid:99671519) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c43e40cf]) (logid:eb844fb5) No inactive management server node found 2025-06-29 04:09:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c43e40cf]) (logid:eb844fb5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:34,079 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2c0c0dfb]) (logid:697f0d8d) HA health check task is running... 2025-06-29 04:09:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9635415b]) (logid:fd04f96b) No inactive management server node found 2025-06-29 04:09:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9635415b]) (logid:fd04f96b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:35,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8176c795]) (logid:3a50d94f) No inactive management server node found 2025-06-29 04:09:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8176c795]) (logid:3a50d94f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:37,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95c2ab5f]) (logid:b8c6b165) No inactive management server node found 2025-06-29 04:09:37,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95c2ab5f]) (logid:b8c6b165) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:09:38,080 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-88f1a038]) (logid:0793f593) HA health check task is running... 2025-06-29 04:09:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d44b5fb]) (logid:c4a43b0b) No inactive management server node found 2025-06-29 04:09:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d44b5fb]) (logid:c4a43b0b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:40,293 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8486: Seq 7-8486: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:09:40,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8486 2025-06-29 04:09:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f012395f]) (logid:34df24bb) No inactive management server node found 2025-06-29 04:09:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f012395f]) (logid:34df24bb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03ffd5a0]) (logid:bcc11ec8) No inactive management server node found 2025-06-29 04:09:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03ffd5a0]) (logid:bcc11ec8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:42,081 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-587f4bfc]) (logid:661fe1b6) HA health check task is running... 2025-06-29 04:09:43,042 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-9c00897a]) (logid:47767662) DbCollector is running... 2025-06-29 04:09:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ca780048]) (logid:9e9074b5) Found 0 routers to update status. 2025-06-29 04:09:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ca780048]) (logid:9e9074b5) Found 0 VPC's to update Redundant State. 2025-06-29 04:09:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ca780048]) (logid:9e9074b5) Found 0 networks to update RvR status. 2025-06-29 04:09:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37088bd7]) (logid:56269d06) No inactive management server node found 2025-06-29 04:09:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-37088bd7]) (logid:56269d06) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) ManagementServerCollector is running... 2025-06-29 04:09:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) DB properties were already loaded 2025-06-29 04:09:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:09:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Execution of process [4066224] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:09:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Exit value of process [4066224] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:09:44,057 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Process [4066224] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:09:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) usage install: null 2025-06-29 04:09:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Sessions active: 0 2025-06-29 04:09:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:09:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066227] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:09:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:09:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066232] for command [/bin/bash -c uname -r ]. 2025-06-29 04:09:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:09:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066233] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:09:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) system memory from /proc: 4887515136 2025-06-29 04:09:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:09:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066238] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:09:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) free memory from /proc: 301871104 2025-06-29 04:09:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:09:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066243] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:09:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) used memory from /proc: 1070340 2025-06-29 04:09:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:09:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066244] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:09:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:09:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066245] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:09:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:09:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066246] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:09:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:09:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066253] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:09:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:09:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Successfully executed process [4066254] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail buffersdirect.count, 93 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail buffersdirect.used, 321663 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail buffersmapped.capacity, 0 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail buffersmapped.count, 0 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail buffersmapped.used, 0 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail jvmuptime, 2043831 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memoryheap.committed, 500695040 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memoryheap.init, 77594624 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memoryheap.usage, 0.20317165644016283 2025-06-29 04:09:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorynon-heap.committed, 194969600 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorynon-heap.max, -1 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorynon-heap.usage, -1.92692984E8 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorynon-heap.used, 192692984 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.109375 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21282429933015629 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012379668653011322 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.Metaspace.usage, 0.9934494306456368 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.PS-Eden-Space.usage, 0.28868228221307624 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorytotal.committed, 695664640 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorytotal.init, 85262336 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorytotal.max, 1908932607 2025-06-29 04:09:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail memorytotal.used, 580771232 2025-06-29 04:09:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail threadsdeadlocks, [] 2025-06-29 04:09:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail threadsnew.count, 0 2025-06-29 04:09:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) not storing detail threadstimed_waiting.count, 82 2025-06-29 04:09:44,135 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-dac34f5d]) (logid:aefc4da0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:09:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":113562248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2043760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.47,"totalInit":85262336,"totalUsed":579821912,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":387841000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":301871104,"systemMemoryUsed":1070340,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.47,1.02,0.73],"systemCyclesUsage":[66408490,53830206,4589392243],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:09:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 67, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:09:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":113562248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2043760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.47,"totalInit":85262336,"totalUsed":579821912,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":387841000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":301871104,"systemMemoryUsed":1070340,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.47,1.02,0.73],"systemCyclesUsage":[66408490,53830206,4589392243],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:09:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:09:44,141 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-17:[ctx-aa517e91]) (logid:f77717fb) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:09:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":113562248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2043760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.47,"totalInit":85262336,"totalUsed":579821912,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":387841000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":301871104,"systemMemoryUsed":1070340,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.47,1.02,0.73],"systemCyclesUsage":[66408490,53830206,4589392243],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:09:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:09:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 67, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:09:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":113562248,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2043760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.47,"totalInit":85262336,"totalUsed":579821912,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":387841000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":301871104,"systemMemoryUsed":1070340,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.47,1.02,0.73],"systemCyclesUsage":[66408490,53830206,4589392243],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:09:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cb312df]) (logid:e58a41bb) No inactive management server node found 2025-06-29 04:09:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cb312df]) (logid:e58a41bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:09:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:09:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:09:46,082 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2ac980fc]) (logid:2bdb36fa) HA health check task is running... 2025-06-29 04:09:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82217fa7]) (logid:6ed30724) No inactive management server node found 2025-06-29 04:09:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82217fa7]) (logid:6ed30724) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:46,647 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:09:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:09:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:09:46,652 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:09:47,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:09:47,078 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:09:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:09:47,082 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:09:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a16a0093]) (logid:40d6a673) No inactive management server node found 2025-06-29 04:09:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a16a0093]) (logid:40d6a673) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:48,089 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2d471663]) (logid:9d86823e) Resetting hosts suitable for reconnect 2025-06-29 04:09:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2d471663]) (logid:9d86823e) Completed resetting hosts suitable for reconnect 2025-06-29 04:09:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2d471663]) (logid:9d86823e) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:09:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2d471663]) (logid:9d86823e) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:09:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2d471663]) (logid:9d86823e) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:09:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-2d471663]) (logid:9d86823e) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:09:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1767946c]) (logid:67f9d15f) VM scheduler.poll is being called at 2025-06-29T02:10:00+0000 2025-06-29 04:09:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1767946c]) (logid:67f9d15f) Got 0 scheduled jobs to be executed at 2025-06-29T02:10:00+0000 2025-06-29 04:09:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-1767946c]) (logid:67f9d15f) Cleaned up 0 VM scheduled job entries 2025-06-29 04:09:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ecc0fe08]) (logid:4934bec8) VM Operation Thread Running 2025-06-29 04:09:48,346 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4e2b1f9c]) (logid:54e1dd0b) ClusterDRS.poll is being called at 2025-06-29T02:10:00+0000 2025-06-29 04:09:48,355 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4e2b1f9c]) (logid:54e1dd0b) Removed 0 old drs migration plans 2025-06-29 04:09:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-addc0a7b]) (logid:b2518389) No inactive management server node found 2025-06-29 04:09:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-addc0a7b]) (logid:b2518389) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:50,083 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5b93e151]) (logid:4ca52633) HA health check task is running... 2025-06-29 04:09:50,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8488: Seq 7-8488: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:09:50,307 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8488 2025-06-29 04:09:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c88dbca5]) (logid:b7d0aff1) No inactive management server node found 2025-06-29 04:09:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c88dbca5]) (logid:b7d0aff1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-794ab258]) (logid:f9f682da) No inactive management server node found 2025-06-29 04:09:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-794ab258]) (logid:f9f682da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e681c450]) (logid:0a6a48b6) No inactive management server node found 2025-06-29 04:09:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e681c450]) (logid:0a6a48b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:54,084 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9a844ac5]) (logid:93a7dbda) HA health check task is running... 2025-06-29 04:09:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e67d799]) (logid:d7466cbf) No inactive management server node found 2025-06-29 04:09:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e67d799]) (logid:d7466cbf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4c5e70a]) (logid:eea64501) No inactive management server node found 2025-06-29 04:09:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4c5e70a]) (logid:eea64501) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:58,085 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-496e6a31]) (logid:1af83e5a) HA health check task is running... 2025-06-29 04:09:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d9eec91b]) (logid:4e602242) Enabled non-edge zones available for scan: 1 2025-06-29 04:09:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d9eec91b]) (logid:4e602242) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:09:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d9eec91b]) (logid:4e602242) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:09:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b60301f2]) (logid:852eb095) Enabled non-edge zones available for scan: 1 2025-06-29 04:09:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b60301f2]) (logid:852eb095) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:09:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ad0e067]) (logid:0f0e0b8b) No inactive management server node found 2025-06-29 04:09:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ad0e067]) (logid:0f0e0b8b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:09:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca36915e]) (logid:25917e8a) No inactive management server node found 2025-06-29 04:09:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca36915e]) (logid:25917e8a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:00,294 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8489: Seq 7-8489: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:00,304 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8489 2025-06-29 04:10:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df4ae6cd]) (logid:f3542728) No inactive management server node found 2025-06-29 04:10:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df4ae6cd]) (logid:f3542728) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:02,086 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8460996b]) (logid:a2a608ed) HA health check task is running... 2025-06-29 04:10:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbcce93e]) (logid:0ab6af0f) No inactive management server node found 2025-06-29 04:10:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbcce93e]) (logid:0ab6af0f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae29a113]) (logid:490cb383) No inactive management server node found 2025-06-29 04:10:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae29a113]) (logid:490cb383) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f656a2f5]) (logid:2fc0ead8) No inactive management server node found 2025-06-29 04:10:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f656a2f5]) (logid:2fc0ead8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:06,087 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3bd8ed49]) (logid:6f806998) HA health check task is running... 2025-06-29 04:10:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee072bd7]) (logid:a60012d7) No inactive management server node found 2025-06-29 04:10:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee072bd7]) (logid:a60012d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-668a9de4]) (logid:d6610ac3) No inactive management server node found 2025-06-29 04:10:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-668a9de4]) (logid:d6610ac3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:10,088 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-90fb9949]) (logid:4de03ab1) HA health check task is running... 2025-06-29 04:10:10,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8490: Seq 7-8490: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:10,305 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8490 2025-06-29 04:10:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d93a0e9]) (logid:600aa83f) No inactive management server node found 2025-06-29 04:10:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d93a0e9]) (logid:600aa83f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61a84585]) (logid:413fe553) No inactive management server node found 2025-06-29 04:10:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61a84585]) (logid:413fe553) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-67b55677]) (logid:1e920c62) Found 0 routers to update status. 2025-06-29 04:10:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-67b55677]) (logid:1e920c62) Found 0 VPC's to update Redundant State. 2025-06-29 04:10:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-67b55677]) (logid:1e920c62) Found 0 networks to update RvR status. 2025-06-29 04:10:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62b9aefe]) (logid:2f5caeaf) No inactive management server node found 2025-06-29 04:10:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62b9aefe]) (logid:2f5caeaf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:14,089 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a088ce37]) (logid:8bbc183d) HA health check task is running... 2025-06-29 04:10:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06b4e2b9]) (logid:7e54493f) No inactive management server node found 2025-06-29 04:10:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06b4e2b9]) (logid:7e54493f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f2d9053]) (logid:5dce3dc7) No inactive management server node found 2025-06-29 04:10:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f2d9053]) (logid:5dce3dc7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62a84b3a]) (logid:ecd7124e) No inactive management server node found 2025-06-29 04:10:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62a84b3a]) (logid:ecd7124e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:17,993 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-ed25241f]) (logid:a9e9c7e3) AutoScaling Monitor is running... 2025-06-29 04:10:18,054 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-9dd4bbb6]) (logid:b17ffd5b) Starting to remove all volume_stats rows older than [Sat Jun 28 16:10:18 CEST 2025]. 2025-06-29 04:10:18,055 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-9dd4bbb6]) (logid:b17ffd5b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:10:18 CEST 2025]. 2025-06-29 04:10:18,090 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6e34d2b8]) (logid:639c1b30) HA health check task is running... 2025-06-29 04:10:18,183 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-efc9417c]) (logid:d6c03ee4) Starting to remove all vm_stats rows older than [Sat Jun 28 16:10:18 CEST 2025]. 2025-06-29 04:10:18,206 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-efc9417c]) (logid:d6c03ee4) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:10:18 CEST 2025]. 2025-06-29 04:10:18,403 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:10:18,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:10:18,406 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) Seq 2-4725683383995203732: Routed from 90520738888109 2025-06-29 04:10:18,450 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) Seq 1-4725683383995203732: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:10:18,451 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:10:18,452 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) Seq 2-3926857400090362013: Routed from 90520738888109 2025-06-29 04:10:18,497 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-818808b5]) (logid:d68854b8) Seq 2-3926857400090362013: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:10:19,079 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:10:19,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:10:19,084 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) Seq 2-4725683383995203733: Routed from 90520738888109 2025-06-29 04:10:19,140 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) Seq 1-4725683383995203733: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:10:19,175 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:10:19,176 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) Seq 2-3926857400090362014: Routed from 90520738888109 2025-06-29 04:10:19,291 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-e6849c3d]) (logid:baa3b0d8) Seq 2-3926857400090362014: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:10:19,351 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) StorageCollector is running... 2025-06-29 04:10:19,357 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:10:19,357 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:10:19,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:10:19,358 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 2-3938116399158788137: Routed from 90520738888109 2025-06-29 04:10:19,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9af60d67]) (logid:4d4f6e22) No inactive management server node found 2025-06-29 04:10:19,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9af60d67]) (logid:4d4f6e22) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:19,419 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 4-3938116399158788137: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:10:19,423 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:10:19,423 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:10:19,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:10:19,424 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 2-4725683383995203734: Routed from 90520738888109 2025-06-29 04:10:19,472 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 1-4725683383995203734: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:10:19,483 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:10:19,483 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:10:19,483 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:10:19,484 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 2-4725683383995203735: Routed from 90520738888109 2025-06-29 04:10:19,573 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 1-4725683383995203735: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:10:19,584 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:10:19,584 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:10:19,584 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:10:19,586 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 2-4725683383995203736: Routed from 90520738888109 2025-06-29 04:10:19,670 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 1-4725683383995203736: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:10:19,673 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:10:19,673 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:10:19,673 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:10:19,674 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 2-3926857400090362015: Routed from 90520738888109 2025-06-29 04:10:19,678 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8a67b5b2]) (logid:c11750c4) Seq 2-3926857400090362015: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:10:20,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8491: Seq 7-8491: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:20,305 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8491 2025-06-29 04:10:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b161ed00]) (logid:9591b151) No inactive management server node found 2025-06-29 04:10:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b161ed00]) (logid:9591b151) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:22,090 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4669c115]) (logid:6c65e2ef) HA health check task is running... 2025-06-29 04:10:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df470095]) (logid:c3b5bec1) No inactive management server node found 2025-06-29 04:10:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df470095]) (logid:c3b5bec1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3960129]) (logid:582e6e94) No inactive management server node found 2025-06-29 04:10:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3960129]) (logid:582e6e94) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:25,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8492: Seq 7-8492: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:25,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8492 2025-06-29 04:10:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81ce0b3c]) (logid:e32dc653) No inactive management server node found 2025-06-29 04:10:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81ce0b3c]) (logid:e32dc653) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:26,091 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-28198248]) (logid:c4d28a1a) HA health check task is running... 2025-06-29 04:10:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7caef26d]) (logid:a8ea8dca) No inactive management server node found 2025-06-29 04:10:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7caef26d]) (logid:a8ea8dca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f43043e7]) (logid:dd368774) Enabled non-edge zones available for scan: 1 2025-06-29 04:10:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f43043e7]) (logid:dd368774) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:10:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f43043e7]) (logid:dd368774) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:10:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-780e020e]) (logid:88b2ebdd) Enabled non-edge zones available for scan: 1 2025-06-29 04:10:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-780e020e]) (logid:88b2ebdd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:10:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b6b6d54]) (logid:aef3d876) No inactive management server node found 2025-06-29 04:10:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b6b6d54]) (logid:aef3d876) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea1ced93]) (logid:45a27ca6) No inactive management server node found 2025-06-29 04:10:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea1ced93]) (logid:45a27ca6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:30,092 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f94d145d]) (logid:c881f954) HA health check task is running... 2025-06-29 04:10:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35c012a6]) (logid:13707a03) No inactive management server node found 2025-06-29 04:10:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35c012a6]) (logid:13707a03) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf7671d9]) (logid:a2814c0d) No inactive management server node found 2025-06-29 04:10:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf7671d9]) (logid:a2814c0d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:34,093 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6281eefc]) (logid:239b00ed) HA health check task is running... 2025-06-29 04:10:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb2699dd]) (logid:1360981b) No inactive management server node found 2025-06-29 04:10:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb2699dd]) (logid:1360981b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:35,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8493: Seq 7-8493: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:35,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8493 2025-06-29 04:10:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e559134c]) (logid:566b1326) No inactive management server node found 2025-06-29 04:10:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e559134c]) (logid:566b1326) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca41fdba]) (logid:c48c7143) No inactive management server node found 2025-06-29 04:10:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca41fdba]) (logid:c48c7143) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:38,094 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e8567c43]) (logid:ee8b3897) HA health check task is running... 2025-06-29 04:10:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5672a3dd]) (logid:85142e0c) No inactive management server node found 2025-06-29 04:10:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5672a3dd]) (logid:85142e0c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63e2d79e]) (logid:263e699f) No inactive management server node found 2025-06-29 04:10:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63e2d79e]) (logid:263e699f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-473bd529]) (logid:5fdafe6f) No inactive management server node found 2025-06-29 04:10:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-473bd529]) (logid:5fdafe6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:42,095 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4848edf1]) (logid:ecfda92d) HA health check task is running... 2025-06-29 04:10:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-ef52a521]) (logid:cfc0ad62) DbCollector is running... 2025-06-29 04:10:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8ac56fd1]) (logid:ed6e167f) Found 0 routers to update status. 2025-06-29 04:10:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8ac56fd1]) (logid:ed6e167f) Found 0 VPC's to update Redundant State. 2025-06-29 04:10:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8ac56fd1]) (logid:ed6e167f) Found 0 networks to update RvR status. 2025-06-29 04:10:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0d5e16a]) (logid:c7b3d09c) No inactive management server node found 2025-06-29 04:10:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0d5e16a]) (logid:c7b3d09c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) ManagementServerCollector is running... 2025-06-29 04:10:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) DB properties were already loaded 2025-06-29 04:10:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:10:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Execution of process [4066267] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:10:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Exit value of process [4066267] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:10:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Process [4066267] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:10:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) usage install: null 2025-06-29 04:10:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Sessions active: 0 2025-06-29 04:10:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:10:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066270] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:10:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:10:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066275] for command [/bin/bash -c uname -r ]. 2025-06-29 04:10:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:10:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066276] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:10:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) system memory from /proc: 4887515136 2025-06-29 04:10:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:10:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066281] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:10:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) free memory from /proc: 290004992 2025-06-29 04:10:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:10:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066286] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:10:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) used memory from /proc: 1070468 2025-06-29 04:10:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:10:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066287] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:10:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:10:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066288] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:10:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:10:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066289] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:10:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:10:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066296] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:10:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:10:44,131 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Successfully executed process [4066297] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail buffersdirect.count, 93 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail buffersdirect.used, 321663 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail buffersmapped.capacity, 0 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail buffersmapped.count, 0 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail buffersmapped.used, 0 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail jvmuptime, 2103837 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memoryheap.committed, 500695040 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memoryheap.init, 77594624 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memoryheap.usage, 0.21483197797624923 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorynon-heap.committed, 195166208 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorynon-heap.max, -1 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorynon-heap.usage, -1.92900024E8 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorynon-heap.used, 192900024 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.10962180918421754 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21385635018495686 2025-06-29 04:10:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.Compressed-Class-Space.usage, 0.012379668653011322 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.Metaspace.usage, 0.9933084058963932 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.PS-Eden-Space.usage, 0.3228133547896781 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorytotal.committed, 695861248 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorytotal.init, 85262336 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorytotal.max, 1908932607 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail memorytotal.used, 602999792 2025-06-29 04:10:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail threadsdeadlocks, [] 2025-06-29 04:10:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail threadsnew.count, 0 2025-06-29 04:10:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:10:44,138 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-e2bbb93c]) (logid:2d902c44) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:10:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":91283144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":602188624,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":410099768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":290004992,"systemMemoryUsed":1070468,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.95,0.72],"systemCyclesUsage":[66408852,53830514,4589415331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:10:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 69, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:10:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":91283144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":602188624,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":410099768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":290004992,"systemMemoryUsed":1070468,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.95,0.72],"systemCyclesUsage":[66408852,53830514,4589415331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:10:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:10:44,147 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:10:44,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 69, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:10:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":91283144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":602188624,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":410099768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":290004992,"systemMemoryUsed":1070468,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.95,0.72],"systemCyclesUsage":[66408852,53830514,4589415331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:10:44,149 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-18:[ctx-fb482ff7]) (logid:943ae83b) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:10:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":91283144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.56,"totalInit":85262336,"totalUsed":602188624,"totalCommitted":695664640,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":410099768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":290004992,"systemMemoryUsed":1070468,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.56,0.95,0.72],"systemCyclesUsage":[66408852,53830514,4589415331],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:10:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec013e40]) (logid:14a92ecc) No inactive management server node found 2025-06-29 04:10:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec013e40]) (logid:14a92ecc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:45,296 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8494: Seq 7-8494: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:45,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8494 2025-06-29 04:10:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:10:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:10:46,096 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-19e83a13]) (logid:faca3365) HA health check task is running... 2025-06-29 04:10:46,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f11e346d]) (logid:d8bbf36d) No inactive management server node found 2025-06-29 04:10:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f11e346d]) (logid:d8bbf36d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms 2025-06-29 04:10:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:10:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:10:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:10:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:10:47,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:10:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:10:47,100 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:10:47,102 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:10:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa0038ff]) (logid:39b1fd05) No inactive management server node found 2025-06-29 04:10:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa0038ff]) (logid:39b1fd05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9aa66f57]) (logid:66f0c9ac) VM scheduler.poll is being called at 2025-06-29T02:11:00+0000 2025-06-29 04:10:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9aa66f57]) (logid:66f0c9ac) Got 0 scheduled jobs to be executed at 2025-06-29T02:11:00+0000 2025-06-29 04:10:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9aa66f57]) (logid:66f0c9ac) Cleaned up 0 VM scheduled job entries 2025-06-29 04:10:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-718fbdaa]) (logid:47c1ccad) VM Operation Thread Running 2025-06-29 04:10:48,346 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-29e03b14]) (logid:57ece407) ClusterDRS.poll is being called at 2025-06-29T02:11:00+0000 2025-06-29 04:10:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-29e03b14]) (logid:57ece407) Removed 0 old drs migration plans 2025-06-29 04:10:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-506b137a]) (logid:9f09349a) No inactive management server node found 2025-06-29 04:10:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-506b137a]) (logid:9f09349a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:50,097 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8388d582]) (logid:db2d5cd1) HA health check task is running... 2025-06-29 04:10:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce844491]) (logid:c514e789) No inactive management server node found 2025-06-29 04:10:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce844491]) (logid:c514e789) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-865a2e8c]) (logid:526b3bf3) No inactive management server node found 2025-06-29 04:10:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-865a2e8c]) (logid:526b3bf3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:10:53,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0ef0f42]) (logid:1f443518) No inactive management server node found 2025-06-29 04:10:53,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0ef0f42]) (logid:1f443518) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:54,098 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-96345d79]) (logid:1da964f3) HA health check task is running... 2025-06-29 04:10:55,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8496: Seq 7-8496: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:10:55,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8496 2025-06-29 04:10:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96f33b57]) (logid:7fcf637b) No inactive management server node found 2025-06-29 04:10:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96f33b57]) (logid:7fcf637b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf4e98f1]) (logid:889b0afa) No inactive management server node found 2025-06-29 04:10:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf4e98f1]) (logid:889b0afa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-faa77a09]) (logid:33cd721d) Enabled non-edge zones available for scan: 1 2025-06-29 04:10:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-faa77a09]) (logid:33cd721d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:10:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-faa77a09]) (logid:33cd721d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:10:58,099 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3d7b36c3]) (logid:e6f4f90d) HA health check task is running... 2025-06-29 04:10:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-09f9148f]) (logid:ea41bb50) Enabled non-edge zones available for scan: 1 2025-06-29 04:10:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-09f9148f]) (logid:ea41bb50) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:10:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9381513b]) (logid:a62a6399) No inactive management server node found 2025-06-29 04:10:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9381513b]) (logid:a62a6399) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:10:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfe48895]) (logid:f43b21ab) No inactive management server node found 2025-06-29 04:10:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfe48895]) (logid:f43b21ab) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af576324]) (logid:7ea3a36e) No inactive management server node found 2025-06-29 04:11:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af576324]) (logid:7ea3a36e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:02,100 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3debf2c2]) (logid:2e8c6e68) HA health check task is running... 2025-06-29 04:11:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4087811]) (logid:8b93377c) No inactive management server node found 2025-06-29 04:11:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4087811]) (logid:8b93377c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b98a84c2]) (logid:bf535246) No inactive management server node found 2025-06-29 04:11:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b98a84c2]) (logid:bf535246) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:05,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8497: Seq 7-8497: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:05,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8497 2025-06-29 04:11:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c86821]) (logid:d4bb50f5) No inactive management server node found 2025-06-29 04:11:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09c86821]) (logid:d4bb50f5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:06,101 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3a06d6e7]) (logid:66169645) HA health check task is running... 2025-06-29 04:11:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-226f46a7]) (logid:c7324864) No inactive management server node found 2025-06-29 04:11:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-226f46a7]) (logid:c7324864) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:08,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ffe00d3]) (logid:44a09388) No inactive management server node found 2025-06-29 04:11:08,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ffe00d3]) (logid:44a09388) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:10,103 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a3ad9204]) (logid:5de71dae) HA health check task is running... 2025-06-29 04:11:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a95e45e9]) (logid:eee25877) No inactive management server node found 2025-06-29 04:11:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a95e45e9]) (logid:eee25877) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f098376]) (logid:d3adcca7) No inactive management server node found 2025-06-29 04:11:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f098376]) (logid:d3adcca7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9c805ecc]) (logid:02c61d72) Found 0 routers to update status. 2025-06-29 04:11:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9c805ecc]) (logid:02c61d72) Found 0 VPC's to update Redundant State. 2025-06-29 04:11:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9c805ecc]) (logid:02c61d72) Found 0 networks to update RvR status. 2025-06-29 04:11:14,105 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e7e32f89]) (logid:9a784262) HA health check task is running... 2025-06-29 04:11:15,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8498: Seq 7-8498: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:15,645 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e0bb59b]) (logid:bc59008d) No inactive management server node found 2025-06-29 04:11:15,645 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e0bb59b]) (logid:bc59008d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:15,645 DEBUG [c.c.c.ClusterManagerImpl$4] (Cluster-Heartbeat-1:[ctx-6e0bb59b]) (logid:bc59008d) Management server heartbeat takes too long to finish. profiler: Done. Duration: 2297ms, profilerHeartbeatUpdate: Done. Duration: 2294ms, profilerPeerScan: Done. Duration: 2ms 2025-06-29 04:11:15,647 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8498 2025-06-29 04:11:15,659 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09d08f91]) (logid:31638ca4) No inactive management server node found 2025-06-29 04:11:15,659 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09d08f91]) (logid:31638ca4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2cfac04]) (logid:2372eeed) No inactive management server node found 2025-06-29 04:11:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2cfac04]) (logid:2372eeed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f223696a]) (logid:0ad6470b) No inactive management server node found 2025-06-29 04:11:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f223696a]) (logid:0ad6470b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:17,996 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-570c2193]) (logid:bac67ef3) AutoScaling Monitor is running... 2025-06-29 04:11:18,057 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2af7bcc5]) (logid:0d7b2803) Starting to remove all volume_stats rows older than [Sat Jun 28 16:11:18 CEST 2025]. 2025-06-29 04:11:18,058 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2af7bcc5]) (logid:0d7b2803) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:11:18 CEST 2025]. 2025-06-29 04:11:18,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9bc5927f]) (logid:f4f1081c) Resetting hosts suitable for reconnect 2025-06-29 04:11:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9bc5927f]) (logid:f4f1081c) Completed resetting hosts suitable for reconnect 2025-06-29 04:11:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9bc5927f]) (logid:f4f1081c) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:11:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9bc5927f]) (logid:f4f1081c) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:11:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9bc5927f]) (logid:f4f1081c) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:11:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-9bc5927f]) (logid:f4f1081c) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:11:18,106 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8dbbdcfa]) (logid:8478a861) HA health check task is running... 2025-06-29 04:11:18,208 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-8111b3d1]) (logid:60064493) Starting to remove all vm_stats rows older than [Sat Jun 28 16:11:18 CEST 2025]. 2025-06-29 04:11:18,234 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-8111b3d1]) (logid:60064493) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:11:18 CEST 2025]. 2025-06-29 04:11:18,500 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:11:18,501 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:11:18,503 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) Seq 2-4725683383995203737: Routed from 90520738888109 2025-06-29 04:11:18,547 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) Seq 1-4725683383995203737: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:11:18,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:11:18,549 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) Seq 2-3926857400090362016: Routed from 90520738888109 2025-06-29 04:11:18,594 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5c6c38e3]) (logid:71f5db78) Seq 2-3926857400090362016: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:11:19,349 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:11:19,353 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:11:19,354 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) Seq 2-4725683383995203738: Routed from 90520738888109 2025-06-29 04:11:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44f8155d]) (logid:26d02e73) No inactive management server node found 2025-06-29 04:11:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44f8155d]) (logid:26d02e73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:19,409 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) Seq 1-4725683383995203738: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:11:19,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:11:19,443 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) Seq 2-3926857400090362017: Routed from 90520738888109 2025-06-29 04:11:19,529 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-a922079e]) (logid:a1271bf5) Seq 2-3926857400090362017: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:11:19,678 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) StorageCollector is running... 2025-06-29 04:11:19,685 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:11:19,685 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:11:19,685 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:11:19,686 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-3938116399158788138: Routed from 90520738888109 2025-06-29 04:11:19,741 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 4-3938116399158788138: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:11:19,745 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:11:19,745 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:11:19,745 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:11:19,746 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-4725683383995203739: Routed from 90520738888109 2025-06-29 04:11:19,793 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 1-4725683383995203739: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:11:19,804 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:11:19,804 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:11:19,804 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:11:19,805 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-3926857400090362018: Routed from 90520738888109 2025-06-29 04:11:19,817 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-3926857400090362018: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:11:19,828 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:11:19,828 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:11:19,828 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:11:19,829 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-4725683383995203740: Routed from 90520738888109 2025-06-29 04:11:19,913 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 1-4725683383995203740: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:11:19,914 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:11:19,914 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:11:19,914 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:11:19,915 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-3926857400090362019: Routed from 90520738888109 2025-06-29 04:11:19,959 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-8225f6b7]) (logid:2d95a9d4) Seq 2-3926857400090362019: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:11:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4510501b]) (logid:cc150764) No inactive management server node found 2025-06-29 04:11:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4510501b]) (logid:cc150764) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:22,107 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-75d844d2]) (logid:b5944f53) HA health check task is running... 2025-06-29 04:11:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d045e35]) (logid:ab1ac815) No inactive management server node found 2025-06-29 04:11:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d045e35]) (logid:ab1ac815) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aea5ff3c]) (logid:94dc67ce) No inactive management server node found 2025-06-29 04:11:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aea5ff3c]) (logid:94dc67ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:25,258 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8499: Seq 7-8499: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:25,267 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8499 2025-06-29 04:11:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17b9b87b]) (logid:a8aaf31e) No inactive management server node found 2025-06-29 04:11:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17b9b87b]) (logid:a8aaf31e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:26,108 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-746d7fff]) (logid:92cef32c) HA health check task is running... 2025-06-29 04:11:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df76ff35]) (logid:92f2c22a) No inactive management server node found 2025-06-29 04:11:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df76ff35]) (logid:92f2c22a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-093f3134]) (logid:4b603e3e) Enabled non-edge zones available for scan: 1 2025-06-29 04:11:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-093f3134]) (logid:4b603e3e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:11:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-093f3134]) (logid:4b603e3e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:11:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-23e37147]) (logid:336c768c) Enabled non-edge zones available for scan: 1 2025-06-29 04:11:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-23e37147]) (logid:336c768c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:11:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed7973be]) (logid:68475721) No inactive management server node found 2025-06-29 04:11:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed7973be]) (logid:68475721) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e9ebf9a]) (logid:5bdaa4ca) No inactive management server node found 2025-06-29 04:11:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e9ebf9a]) (logid:5bdaa4ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:30,109 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1a4b832b]) (logid:df168851) HA health check task is running... 2025-06-29 04:11:30,298 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8500: Seq 7-8500: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:30,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8500 2025-06-29 04:11:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acc5d8b6]) (logid:6f242d39) No inactive management server node found 2025-06-29 04:11:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acc5d8b6]) (logid:6f242d39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55b16630]) (logid:072c4034) No inactive management server node found 2025-06-29 04:11:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55b16630]) (logid:072c4034) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:34,111 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-27eb60fc]) (logid:f6a8506f) HA health check task is running... 2025-06-29 04:11:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10e7b0e2]) (logid:33096856) No inactive management server node found 2025-06-29 04:11:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10e7b0e2]) (logid:33096856) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30654902]) (logid:edf85e20) No inactive management server node found 2025-06-29 04:11:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30654902]) (logid:edf85e20) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfe08ada]) (logid:ebd19df2) No inactive management server node found 2025-06-29 04:11:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfe08ada]) (logid:ebd19df2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:38,112 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a61729d5]) (logid:a85421fb) HA health check task is running... 2025-06-29 04:11:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a656488b]) (logid:b0bd6350) No inactive management server node found 2025-06-29 04:11:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a656488b]) (logid:b0bd6350) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:40,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8501: Seq 7-8501: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:40,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8501 2025-06-29 04:11:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6528f058]) (logid:5a2c24bd) No inactive management server node found 2025-06-29 04:11:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6528f058]) (logid:5a2c24bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8377f23]) (logid:cce1310f) No inactive management server node found 2025-06-29 04:11:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8377f23]) (logid:cce1310f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:42,113 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3f6a46e7]) (logid:40a7e13f) HA health check task is running... 2025-06-29 04:11:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-b615c3e5]) (logid:3f2be940) DbCollector is running... 2025-06-29 04:11:43,094 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-b83f68f8]) (logid:3c85e8c2) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:11:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-b83f68f8]) (logid:3c85e8c2) Snapshot scheduler is being called at [2025-06-29T02:11:43+0000]. 2025-06-29 04:11:43,096 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-b83f68f8]) (logid:3c85e8c2) There are [0] scheduled snapshots to be executed at [2025-06-29T02:11:43+0000]. 2025-06-29 04:11:43,098 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-8bedad47]) (logid:b8fdd52f) Found 0 running routers. 2025-06-29 04:11:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ac96d6c3]) (logid:ec2a8cc6) Found 0 routers to update status. 2025-06-29 04:11:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ac96d6c3]) (logid:ec2a8cc6) Found 0 VPC's to update Redundant State. 2025-06-29 04:11:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ac96d6c3]) (logid:ec2a8cc6) Found 0 networks to update RvR status. 2025-06-29 04:11:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-5d87cbd6]) (logid:c9d0ccf8) External devices are not used. Skipping external device usage collection 2025-06-29 04:11:43,129 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-b0154c59]) (logid:7e0aeecb) Started resource counters recalculation periodic task. 2025-06-29 04:11:43,133 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-b0154c59]) (logid:7e0aeecb) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:11:43,305 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-3579cea8]) (logid:fe9d6f65) Backup backup.poll is being called at 2025-06-29T02:11:43+0000 2025-06-29 04:11:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1598d46a]) (logid:f281c45a) No inactive management server node found 2025-06-29 04:11:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1598d46a]) (logid:f281c45a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:43,544 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-b0154c59]) (logid:7e0aeecb) Finished resource counters recalculation periodic task. 2025-06-29 04:11:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) ManagementServerCollector is running... 2025-06-29 04:11:44,042 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) DB properties were already loaded 2025-06-29 04:11:44,042 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:11:44,060 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Execution of process [4066303] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:11:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Exit value of process [4066303] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:11:44,060 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Process [4066303] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:11:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) usage install: null 2025-06-29 04:11:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Sessions active: 0 2025-06-29 04:11:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:11:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066306] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:11:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:11:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066311] for command [/bin/bash -c uname -r ]. 2025-06-29 04:11:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:11:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066312] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:11:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) system memory from /proc: 4887515136 2025-06-29 04:11:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:11:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066317] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:11:44,086 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) free memory from /proc: 296845312 2025-06-29 04:11:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:11:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066322] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:11:44,099 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) used memory from /proc: 1070724 2025-06-29 04:11:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:11:44,104 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066323] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:11:44,104 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:11:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066324] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:11:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:11:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066325] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:11:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:11:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066332] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:11:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:11:44,132 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Successfully executed process [4066333] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:11:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:11:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail buffersdirect.count, 93 2025-06-29 04:11:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail buffersdirect.used, 321663 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail buffersmapped.capacity, 0 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail buffersmapped.count, 0 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail buffersmapped.used, 0 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail jvmuptime, 2163838 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memoryheap.committed, 500695040 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memoryheap.init, 77594624 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memoryheap.usage, 0.23219992059562533 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorynon-heap.committed, 195297280 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorynon-heap.max, -1 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorynon-heap.usage, -1.93062616E8 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorynon-heap.used, 193062616 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11030391895494535 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2144093461525644 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.Compressed-Class-Space.usage, 0.012379668653011322 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.Metaspace.usage, 0.9933859875942952 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.PS-Eden-Space.usage, 0.3741835246330156 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorytotal.committed, 695992320 2025-06-29 04:11:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorytotal.init, 85262336 2025-06-29 04:11:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorytotal.max, 1908932607 2025-06-29 04:11:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail memorytotal.used, 636316616 2025-06-29 04:11:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail threadsdeadlocks, [] 2025-06-29 04:11:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail threadsnew.count, 0 2025-06-29 04:11:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:11:44,139 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-281c30a8]) (logid:cfee0c8a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:11:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":58196752,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2163764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":635558856,"totalCommitted":695992320,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":443254000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":296845312,"systemMemoryUsed":1070724,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.81,0.69],"systemCyclesUsage":[66409241,53830820,4589438387],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:11:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 71, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:11:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":58196752,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2163764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":635558856,"totalCommitted":695992320,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":443254000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":296845312,"systemMemoryUsed":1070724,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.81,0.69],"systemCyclesUsage":[66409241,53830820,4589438387],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:11:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:11:44,145 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-18:[ctx-9e1a50e3]) (logid:f414f691) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:11:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":58196752,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2163764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":635558856,"totalCommitted":695992320,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":443254000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":296845312,"systemMemoryUsed":1070724,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.81,0.69],"systemCyclesUsage":[66409241,53830820,4589438387],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:11:44,146 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:11:44,146 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 71, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:11:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":58196752,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2163764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.31,"totalInit":85262336,"totalUsed":635558856,"totalCommitted":695992320,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":443254000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":296845312,"systemMemoryUsed":1070724,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.31,0.81,0.69],"systemCyclesUsage":[66409241,53830820,4589438387],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:11:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7af64398]) (logid:259313ec) No inactive management server node found 2025-06-29 04:11:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7af64398]) (logid:259313ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:11:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:11:46,115 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f1843fec]) (logid:ae1c7587) HA health check task is running... 2025-06-29 04:11:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c03ff6ca]) (logid:9af896a4) No inactive management server node found 2025-06-29 04:11:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c03ff6ca]) (logid:9af896a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:11:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:11:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:11:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:11:47,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:11:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:11:47,100 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:11:47,103 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:11:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-109ad577]) (logid:46be2f2f) No inactive management server node found 2025-06-29 04:11:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-109ad577]) (logid:46be2f2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:11:48,047 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-2:[ctx-b838a1ab]) (logid:11a8074e) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:11:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-75d30b0b]) (logid:d00a95ec) VM scheduler.poll is being called at 2025-06-29T02:12:00+0000 2025-06-29 04:11:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-75d30b0b]) (logid:d00a95ec) Got 0 scheduled jobs to be executed at 2025-06-29T02:12:00+0000 2025-06-29 04:11:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-75d30b0b]) (logid:d00a95ec) Cleaned up 0 VM scheduled job entries 2025-06-29 04:11:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-e055ef2a]) (logid:2f3796da) VM Operation Thread Running 2025-06-29 04:11:48,346 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-35fdb71e]) (logid:f155b789) ClusterDRS.poll is being called at 2025-06-29T02:12:00+0000 2025-06-29 04:11:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-35fdb71e]) (logid:f155b789) Removed 0 old drs migration plans 2025-06-29 04:11:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-403a6a59]) (logid:a5ea0108) No inactive management server node found 2025-06-29 04:11:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-403a6a59]) (logid:a5ea0108) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:50,116 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-04b171d2]) (logid:aeb82128) HA health check task is running... 2025-06-29 04:11:50,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8503: Seq 7-8503: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:50,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8503 2025-06-29 04:11:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f47229ed]) (logid:ca167346) No inactive management server node found 2025-06-29 04:11:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f47229ed]) (logid:ca167346) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b23d1db]) (logid:11d21855) No inactive management server node found 2025-06-29 04:11:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b23d1db]) (logid:11d21855) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:53,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d3c2474]) (logid:ce612684) No inactive management server node found 2025-06-29 04:11:53,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d3c2474]) (logid:ce612684) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:54,117 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e33801b5]) (logid:a5fb8cb5) HA health check task is running... 2025-06-29 04:11:55,299 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8504: Seq 7-8504: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:11:55,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8504 2025-06-29 04:11:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b900cb97]) (logid:6b3661af) No inactive management server node found 2025-06-29 04:11:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b900cb97]) (logid:6b3661af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9a6e08c]) (logid:b4e6357a) No inactive management server node found 2025-06-29 04:11:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9a6e08c]) (logid:b4e6357a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d14f31c6]) (logid:dc84d01f) Enabled non-edge zones available for scan: 1 2025-06-29 04:11:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d14f31c6]) (logid:dc84d01f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:11:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d14f31c6]) (logid:dc84d01f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:11:58,118 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-623a806e]) (logid:6fccd22f) HA health check task is running... 2025-06-29 04:11:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3610ac53]) (logid:a125a80d) Enabled non-edge zones available for scan: 1 2025-06-29 04:11:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3610ac53]) (logid:a125a80d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:11:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f850989f]) (logid:345ed7c9) No inactive management server node found 2025-06-29 04:11:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f850989f]) (logid:345ed7c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:11:59,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17130f15]) (logid:d741e7fc) No inactive management server node found 2025-06-29 04:11:59,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17130f15]) (logid:d741e7fc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddb76101]) (logid:460771e8) No inactive management server node found 2025-06-29 04:12:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddb76101]) (logid:460771e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:02,120 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d6569155]) (logid:166af132) HA health check task is running... 2025-06-29 04:12:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f9ad818]) (logid:e79eab47) No inactive management server node found 2025-06-29 04:12:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f9ad818]) (logid:e79eab47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64d84374]) (logid:0bf063f9) No inactive management server node found 2025-06-29 04:12:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64d84374]) (logid:0bf063f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:05,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8505: Seq 7-8505: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:12:05,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8505 2025-06-29 04:12:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c5e29c5]) (logid:eb271774) No inactive management server node found 2025-06-29 04:12:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c5e29c5]) (logid:eb271774) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:06,121 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-2ba833c1]) (logid:ce82d974) HA health check task is running... 2025-06-29 04:12:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fdecc265]) (logid:5b1aa275) No inactive management server node found 2025-06-29 04:12:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fdecc265]) (logid:5b1aa275) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaf51e03]) (logid:417db730) No inactive management server node found 2025-06-29 04:12:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaf51e03]) (logid:417db730) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:10,122 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-30af6877]) (logid:244a46dd) HA health check task is running... 2025-06-29 04:12:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3be0ea0c]) (logid:dbb6d89c) No inactive management server node found 2025-06-29 04:12:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3be0ea0c]) (logid:dbb6d89c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a40805d]) (logid:e925adaf) No inactive management server node found 2025-06-29 04:12:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a40805d]) (logid:e925adaf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:13,004 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-80279c3c]) (logid:33862263) Checking if any host reservation can be released ... 2025-06-29 04:12:13,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-80279c3c]) (logid:33862263) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-80279c3c]) (logid:33862263) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-80279c3c]) (logid:33862263) Done running HostReservationReleaseChecker ... 2025-06-29 04:12:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-56fdc03c]) (logid:5237ab56) Found 0 routers to update status. 2025-06-29 04:12:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-56fdc03c]) (logid:5237ab56) Found 0 VPC's to update Redundant State. 2025-06-29 04:12:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-56fdc03c]) (logid:5237ab56) Found 0 networks to update RvR status. 2025-06-29 04:12:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Running Capacity Checker ... 2025-06-29 04:12:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) recalculating system capacity 2025-06-29 04:12:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Executing cpu/ram capacity update 2025-06-29 04:12:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:12:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:12:13,192 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:12:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:13,205 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:13,213 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:13,215 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:12:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:12:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-26-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:12:13,225 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Done executing cpu/ram capacity update 2025-06-29 04:12:13,225 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Executing storage capacity update 2025-06-29 04:12:13,238 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:12:13,238 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:12:13,238 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:12:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:12:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:12:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:12:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:12:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:12:13,257 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:12:13,258 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:12:13,258 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:12:13,263 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:12:13,271 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:12:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:12:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:12:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-27-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:12:13,278 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Done executing storage capacity update 2025-06-29 04:12:13,278 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Executing capacity updates for public ip and Vlans 2025-06-29 04:12:13,297 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Done capacity updates for public ip and Vlans 2025-06-29 04:12:13,297 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Executing capacity updates for private ip 2025-06-29 04:12:13,303 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Done executing capacity updates for private ip 2025-06-29 04:12:13,303 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Done recalculating system capacity 2025-06-29 04:12:13,347 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-85a7c5fe]) (logid:13f10795) Done running Capacity Checker ... 2025-06-29 04:12:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfd21ce7]) (logid:f1f16355) No inactive management server node found 2025-06-29 04:12:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfd21ce7]) (logid:f1f16355) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:14,124 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-128b7464]) (logid:464b8fbc) HA health check task is running... 2025-06-29 04:12:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce80e728]) (logid:b75f8d06) No inactive management server node found 2025-06-29 04:12:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce80e728]) (logid:b75f8d06) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:15,300 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8506: Seq 7-8506: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:12:15,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8506 2025-06-29 04:12:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c2f058]) (logid:41aa8179) No inactive management server node found 2025-06-29 04:12:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c2f058]) (logid:41aa8179) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc68c545]) (logid:0e5ed220) No inactive management server node found 2025-06-29 04:12:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc68c545]) (logid:0e5ed220) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:12:17,998 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-f51c1dae]) (logid:eaff343b) AutoScaling Monitor is running... 2025-06-29 04:12:18,060 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-3e59e5c3]) (logid:cf3ae98b) Starting to remove all volume_stats rows older than [Sat Jun 28 16:12:18 CEST 2025]. 2025-06-29 04:12:18,061 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-3e59e5c3]) (logid:cf3ae98b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:12:18 CEST 2025]. 2025-06-29 04:12:18,125 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e6407747]) (logid:571efd8c) HA health check task is running... 2025-06-29 04:12:18,235 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-b58be81e]) (logid:d6d7e1d7) Starting to remove all vm_stats rows older than [Sat Jun 28 16:12:18 CEST 2025]. 2025-06-29 04:12:18,265 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-b58be81e]) (logid:d6d7e1d7) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:12:18 CEST 2025]. 2025-06-29 04:12:18,597 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:12:18,598 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:12:18,600 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) Seq 2-4725683383995203741: Routed from 90520738888109 2025-06-29 04:12:18,644 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) Seq 1-4725683383995203741: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:12:18,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:12:18,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) Seq 2-3926857400090362020: Routed from 90520738888109 2025-06-29 04:12:18,690 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bb9e2f18]) (logid:77cdb8ad) Seq 2-3926857400090362020: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:12:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d72ee44f]) (logid:a0a3da3c) No inactive management server node found 2025-06-29 04:12:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d72ee44f]) (logid:a0a3da3c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:19,585 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:12:19,588 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:12:19,589 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) Seq 2-4725683383995203742: Routed from 90520738888109 2025-06-29 04:12:19,642 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) Seq 1-4725683383995203742: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:12:19,676 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:12:19,677 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) Seq 2-3926857400090362021: Routed from 90520738888109 2025-06-29 04:12:19,765 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-065866fc]) (logid:de576083) Seq 2-3926857400090362021: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:12:19,960 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) StorageCollector is running... 2025-06-29 04:12:19,967 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:12:19,967 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:12:19,967 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:12:19,968 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-3938116399158788139: Routed from 90520738888109 2025-06-29 04:12:20,021 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 4-3938116399158788139: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:12:20,025 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:12:20,025 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:12:20,026 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:12:20,027 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-3926857400090362022: Routed from 90520738888109 2025-06-29 04:12:20,042 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-3926857400090362022: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:12:20,052 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:12:20,052 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:12:20,053 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:12:20,054 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-4725683383995203743: Routed from 90520738888109 2025-06-29 04:12:20,102 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 1-4725683383995203743: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:12:20,112 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:12:20,112 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:12:20,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:12:20,114 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-4725683383995203744: Routed from 90520738888109 2025-06-29 04:12:20,197 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 1-4725683383995203744: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:12:20,199 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:12:20,199 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:12:20,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:12:20,201 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-3926857400090362023: Routed from 90520738888109 2025-06-29 04:12:20,245 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-2b810035]) (logid:a052a223) Seq 2-3926857400090362023: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:12:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a63e113e]) (logid:78fbbddf) No inactive management server node found 2025-06-29 04:12:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a63e113e]) (logid:78fbbddf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:22,126 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3cf04191]) (logid:e2f3517e) HA health check task is running... 2025-06-29 04:12:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f066a6f]) (logid:369270ce) No inactive management server node found 2025-06-29 04:12:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f066a6f]) (logid:369270ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a4bcc9b]) (logid:b2c24e88) No inactive management server node found 2025-06-29 04:12:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a4bcc9b]) (logid:b2c24e88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:25,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8507: Seq 7-8507: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:12:25,312 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8507 2025-06-29 04:12:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fa864a7]) (logid:3d4adef8) No inactive management server node found 2025-06-29 04:12:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4fa864a7]) (logid:3d4adef8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:26,127 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b5b4ac21]) (logid:8d16243a) HA health check task is running... 2025-06-29 04:12:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fc412a2]) (logid:a1e57afe) No inactive management server node found 2025-06-29 04:12:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fc412a2]) (logid:a1e57afe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-93ae8524]) (logid:f6e54b68) Enabled non-edge zones available for scan: 1 2025-06-29 04:12:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-93ae8524]) (logid:f6e54b68) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:12:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-93ae8524]) (logid:f6e54b68) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:12:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ed12fd43]) (logid:40ec8b79) Enabled non-edge zones available for scan: 1 2025-06-29 04:12:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ed12fd43]) (logid:40ec8b79) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:12:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36525e43]) (logid:461f3962) No inactive management server node found 2025-06-29 04:12:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36525e43]) (logid:461f3962) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca480825]) (logid:01700409) No inactive management server node found 2025-06-29 04:12:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca480825]) (logid:01700409) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:30,129 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b51e09aa]) (logid:0eb93d8b) HA health check task is running... 2025-06-29 04:12:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26162070]) (logid:fac777a6) No inactive management server node found 2025-06-29 04:12:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26162070]) (logid:fac777a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f8f5702]) (logid:8d5adbba) No inactive management server node found 2025-06-29 04:12:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f8f5702]) (logid:8d5adbba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:34,130 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c8d6809e]) (logid:c6813094) HA health check task is running... 2025-06-29 04:12:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-519b498c]) (logid:c9a149e1) No inactive management server node found 2025-06-29 04:12:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-519b498c]) (logid:c9a149e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:35,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8508: Seq 7-8508: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:12:35,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8508 2025-06-29 04:12:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5894df4c]) (logid:2ce66b66) No inactive management server node found 2025-06-29 04:12:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5894df4c]) (logid:2ce66b66) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55aad4b8]) (logid:5377dafc) No inactive management server node found 2025-06-29 04:12:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55aad4b8]) (logid:5377dafc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:38,132 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-762977ae]) (logid:f61af3c8) HA health check task is running... 2025-06-29 04:12:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c7895b4]) (logid:514c2896) No inactive management server node found 2025-06-29 04:12:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c7895b4]) (logid:514c2896) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96cd6e01]) (logid:77580d1c) No inactive management server node found 2025-06-29 04:12:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96cd6e01]) (logid:77580d1c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e9a2ca4]) (logid:36d86280) No inactive management server node found 2025-06-29 04:12:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e9a2ca4]) (logid:36d86280) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:42,133 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-eb7e3c81]) (logid:ad883360) HA health check task is running... 2025-06-29 04:12:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-6d88c81d]) (logid:686b8fb3) DbCollector is running... 2025-06-29 04:12:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-59ff41f9]) (logid:031278a6) Found 0 routers to update status. 2025-06-29 04:12:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-59ff41f9]) (logid:031278a6) Found 0 VPC's to update Redundant State. 2025-06-29 04:12:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-59ff41f9]) (logid:031278a6) Found 0 networks to update RvR status. 2025-06-29 04:12:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0a1c831]) (logid:f382d13c) No inactive management server node found 2025-06-29 04:12:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0a1c831]) (logid:f382d13c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) ManagementServerCollector is running... 2025-06-29 04:12:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) DB properties were already loaded 2025-06-29 04:12:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:12:44,059 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Execution of process [4066348] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:12:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Exit value of process [4066348] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:12:44,060 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Process [4066348] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:12:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) usage install: null 2025-06-29 04:12:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Sessions active: 0 2025-06-29 04:12:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:12:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066351] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:12:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:12:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066356] for command [/bin/bash -c uname -r ]. 2025-06-29 04:12:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:12:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066357] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:12:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) system memory from /proc: 4887515136 2025-06-29 04:12:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:12:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066362] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:12:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) free memory from /proc: 297852928 2025-06-29 04:12:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:12:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066367] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:12:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) used memory from /proc: 1070852 2025-06-29 04:12:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:12:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066368] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:12:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:12:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066369] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:12:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:12:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066370] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:12:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:12:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066377] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:12:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:12:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Successfully executed process [4066378] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail buffersdirect.count, 93 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail buffersdirect.used, 321663 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail buffersmapped.capacity, 0 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail buffersmapped.count, 0 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail buffersmapped.used, 0 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:12:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail jvmuptime, 2223833 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memoryheap.committed, 500695040 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memoryheap.init, 77594624 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memoryheap.usage, 0.24772849393329657 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorynon-heap.committed, 195362816 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorynon-heap.max, -1 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorynon-heap.usage, -1.93210408E8 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorynon-heap.used, 193210408 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11090375899760065 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21479675575699003 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.Compressed-Class-Space.usage, 0.01238223910331726 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.Metaspace.usage, 0.993556402585013 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.PS-Eden-Space.usage, 0.4201132854148685 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorytotal.committed, 696057856 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorytotal.init, 85262336 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorytotal.max, 1908932607 2025-06-29 04:12:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail memorytotal.used, 666107408 2025-06-29 04:12:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail threadsdeadlocks, [] 2025-06-29 04:12:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail threadsnew.count, 0 2025-06-29 04:12:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:12:44,136 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-80734345]) (logid:6abb70dd) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:12:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":28514432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2223763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.29,"totalInit":85262336,"totalUsed":665391016,"totalCommitted":696057856,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":472897000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297852928,"systemMemoryUsed":1070852,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.29,0.7,0.66],"systemCyclesUsage":[66409623,53831126,4589461450],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:12:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 73, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:12:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":28514432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2223763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.29,"totalInit":85262336,"totalUsed":665391016,"totalCommitted":696057856,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":472897000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297852928,"systemMemoryUsed":1070852,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.29,0.7,0.66],"systemCyclesUsage":[66409623,53831126,4589461450],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:12:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:12:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-18:[ctx-9c32e69e]) (logid:811b4c93) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:12:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":28514432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2223763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.29,"totalInit":85262336,"totalUsed":665391016,"totalCommitted":696057856,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":472897000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297852928,"systemMemoryUsed":1070852,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.29,0.7,0.66],"systemCyclesUsage":[66409623,53831126,4589461450],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:12:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:12:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 73, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:12:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":28514432,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2223763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.29,"totalInit":85262336,"totalUsed":665391016,"totalCommitted":696057856,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":472897000,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297852928,"systemMemoryUsed":1070852,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.29,0.7,0.66],"systemCyclesUsage":[66409623,53831126,4589461450],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:12:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd024133]) (logid:96b4d6d7) No inactive management server node found 2025-06-29 04:12:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd024133]) (logid:96b4d6d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:45,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8509: Seq 7-8509: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:12:45,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8509 2025-06-29 04:12:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:12:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:12:46,135 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-eb2a0529]) (logid:44e652a9) HA health check task is running... 2025-06-29 04:12:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05ed80c7]) (logid:2486dbc0) No inactive management server node found 2025-06-29 04:12:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05ed80c7]) (logid:2486dbc0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:12:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:12:47,073 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:47,075 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:47,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:12:47,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:12:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f304703]) (logid:aa12c0c8) No inactive management server node found 2025-06-29 04:12:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f304703]) (logid:aa12c0c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:48,090 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-67ab0db7]) (logid:f6a9c1bc) Resetting hosts suitable for reconnect 2025-06-29 04:12:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-67ab0db7]) (logid:f6a9c1bc) Completed resetting hosts suitable for reconnect 2025-06-29 04:12:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-67ab0db7]) (logid:f6a9c1bc) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:12:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-67ab0db7]) (logid:f6a9c1bc) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:12:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-67ab0db7]) (logid:f6a9c1bc) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:12:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-67ab0db7]) (logid:f6a9c1bc) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:12:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-119ef64f]) (logid:9b42cfb5) VM scheduler.poll is being called at 2025-06-29T02:13:00+0000 2025-06-29 04:12:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-119ef64f]) (logid:9b42cfb5) Got 0 scheduled jobs to be executed at 2025-06-29T02:13:00+0000 2025-06-29 04:12:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-119ef64f]) (logid:9b42cfb5) Cleaned up 0 VM scheduled job entries 2025-06-29 04:12:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9ae483ee]) (logid:ff455d13) VM Operation Thread Running 2025-06-29 04:12:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d3e9277d]) (logid:45c1014d) ClusterDRS.poll is being called at 2025-06-29T02:13:00+0000 2025-06-29 04:12:48,357 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d3e9277d]) (logid:45c1014d) Removed 0 old drs migration plans 2025-06-29 04:12:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44bbd4ab]) (logid:04bfa43e) No inactive management server node found 2025-06-29 04:12:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44bbd4ab]) (logid:04bfa43e) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:50,136 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ad712d8f]) (logid:097e1988) HA health check task is running... 2025-06-29 04:12:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bade234c]) (logid:88611e45) No inactive management server node found 2025-06-29 04:12:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bade234c]) (logid:88611e45) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71502bc0]) (logid:5aeacfd1) No inactive management server node found 2025-06-29 04:12:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71502bc0]) (logid:5aeacfd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-041d16af]) (logid:9479c90a) No inactive management server node found 2025-06-29 04:12:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-041d16af]) (logid:9479c90a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:54,137 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-179d752f]) (logid:78791be6) HA health check task is running... 2025-06-29 04:12:55,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8511: Seq 7-8511: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:12:55,313 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8511 2025-06-29 04:12:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e9540b3]) (logid:d2ca7968) No inactive management server node found 2025-06-29 04:12:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e9540b3]) (logid:d2ca7968) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d52637a]) (logid:98a3c757) No inactive management server node found 2025-06-29 04:12:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d52637a]) (logid:98a3c757) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c3e8d8e4]) (logid:6ee4b4e9) Enabled non-edge zones available for scan: 1 2025-06-29 04:12:58,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-c3e8d8e4]) (logid:6ee4b4e9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:12:58,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-c3e8d8e4]) (logid:6ee4b4e9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:12:58,139 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b53877c9]) (logid:f10e7a80) HA health check task is running... 2025-06-29 04:12:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4d2b79db]) (logid:f6b54cd2) Enabled non-edge zones available for scan: 1 2025-06-29 04:12:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4d2b79db]) (logid:f6b54cd2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:12:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6256a49]) (logid:ed8c5f24) No inactive management server node found 2025-06-29 04:12:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6256a49]) (logid:ed8c5f24) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:12:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a46582eb]) (logid:d5541157) No inactive management server node found 2025-06-29 04:12:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a46582eb]) (logid:d5541157) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:00,302 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8512: Seq 7-8512: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:00,312 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8512 2025-06-29 04:13:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcd67735]) (logid:a12a614c) No inactive management server node found 2025-06-29 04:13:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcd67735]) (logid:a12a614c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:02,140 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-21358a9f]) (logid:88ed511a) HA health check task is running... 2025-06-29 04:13:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-baf6a6d9]) (logid:e32bd482) No inactive management server node found 2025-06-29 04:13:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-baf6a6d9]) (logid:e32bd482) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d8d304]) (logid:d65e98a4) No inactive management server node found 2025-06-29 04:13:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d8d304]) (logid:d65e98a4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-432fc5a1]) (logid:58f277df) No inactive management server node found 2025-06-29 04:13:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-432fc5a1]) (logid:58f277df) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:06,141 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ce54ae39]) (logid:de8afed9) HA health check task is running... 2025-06-29 04:13:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-798ad647]) (logid:2762501c) No inactive management server node found 2025-06-29 04:13:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-798ad647]) (logid:2762501c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98056d9d]) (logid:df044d2b) No inactive management server node found 2025-06-29 04:13:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98056d9d]) (logid:df044d2b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:10,142 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ef949de4]) (logid:bd247b7b) HA health check task is running... 2025-06-29 04:13:10,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8513: Seq 7-8513: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:10,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8513 2025-06-29 04:13:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-707d64c6]) (logid:b0c69dd5) No inactive management server node found 2025-06-29 04:13:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-707d64c6]) (logid:b0c69dd5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c793a35]) (logid:cdf62e60) No inactive management server node found 2025-06-29 04:13:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c793a35]) (logid:cdf62e60) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b49e5e95]) (logid:9d3f5032) Found 0 routers to update status. 2025-06-29 04:13:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b49e5e95]) (logid:9d3f5032) Found 0 VPC's to update Redundant State. 2025-06-29 04:13:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b49e5e95]) (logid:9d3f5032) Found 0 networks to update RvR status. 2025-06-29 04:13:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-191e0fcc]) (logid:be936a8b) No inactive management server node found 2025-06-29 04:13:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-191e0fcc]) (logid:be936a8b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:14,144 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-62dd40b3]) (logid:48de8260) HA health check task is running... 2025-06-29 04:13:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97554891]) (logid:f22d7857) No inactive management server node found 2025-06-29 04:13:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97554891]) (logid:f22d7857) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50498f69]) (logid:842fa4ea) No inactive management server node found 2025-06-29 04:13:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50498f69]) (logid:842fa4ea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1636270b]) (logid:783a571b) No inactive management server node found 2025-06-29 04:13:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1636270b]) (logid:783a571b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:18,001 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-d35cfb5e]) (logid:a351228e) AutoScaling Monitor is running... 2025-06-29 04:13:18,063 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-cc770275]) (logid:36bfaa5b) Starting to remove all volume_stats rows older than [Sat Jun 28 16:13:18 CEST 2025]. 2025-06-29 04:13:18,063 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-cc770275]) (logid:36bfaa5b) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:13:18 CEST 2025]. 2025-06-29 04:13:18,145 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8b55179a]) (logid:a65f6df9) HA health check task is running... 2025-06-29 04:13:18,267 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-2079a1e6]) (logid:609a308d) Starting to remove all vm_stats rows older than [Sat Jun 28 16:13:18 CEST 2025]. 2025-06-29 04:13:18,295 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-2079a1e6]) (logid:609a308d) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:13:18 CEST 2025]. 2025-06-29 04:13:18,692 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:13:18,694 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:13:18,696 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) Seq 2-4725683383995203745: Routed from 90520738888109 2025-06-29 04:13:18,739 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) Seq 1-4725683383995203745: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:13:18,741 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:13:18,742 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) Seq 2-3926857400090362024: Routed from 90520738888109 2025-06-29 04:13:18,786 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-82d71094]) (logid:18a96316) Seq 2-3926857400090362024: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:13:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e216d6d]) (logid:74a9e84f) No inactive management server node found 2025-06-29 04:13:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e216d6d]) (logid:74a9e84f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:13:19,819 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:13:19,822 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:13:19,823 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) Seq 2-4725683383995203746: Routed from 90520738888109 2025-06-29 04:13:19,876 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) Seq 1-4725683383995203746: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:13:19,910 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:13:19,911 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) Seq 2-3926857400090362025: Routed from 90520738888109 2025-06-29 04:13:20,080 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-9d29ab97]) (logid:2441337b) Seq 2-3926857400090362025: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:13:20,246 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) StorageCollector is running... 2025-06-29 04:13:20,252 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:13:20,252 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:13:20,252 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:13:20,253 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3938116399158788140: Routed from 90520738888109 2025-06-29 04:13:20,305 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8514: Seq 7-8514: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:20,309 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 4-3938116399158788140: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:13:20,314 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:13:20,314 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:13:20,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:13:20,315 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3926857400090362026: Routed from 90520738888109 2025-06-29 04:13:20,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8514 2025-06-29 04:13:20,329 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3926857400090362026: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:13:20,345 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:13:20,345 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:13:20,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:13:20,346 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3926857400090362027: Routed from 90520738888109 2025-06-29 04:13:20,446 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3926857400090362027: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:13:20,459 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:13:20,459 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:13:20,459 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:13:20,461 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-4725683383995203747: Routed from 90520738888109 2025-06-29 04:13:20,505 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 1-4725683383995203747: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:13:20,508 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:13:20,508 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:13:20,508 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:13:20,510 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3926857400090362028: Routed from 90520738888109 2025-06-29 04:13:20,554 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-09414013]) (logid:f1624797) Seq 2-3926857400090362028: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:13:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6adbbc07]) (logid:fc956026) No inactive management server node found 2025-06-29 04:13:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6adbbc07]) (logid:fc956026) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:22,146 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5b0b593d]) (logid:359a2021) HA health check task is running... 2025-06-29 04:13:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f57b30d4]) (logid:b46bcb32) No inactive management server node found 2025-06-29 04:13:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f57b30d4]) (logid:b46bcb32) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7b96d1d]) (logid:774460a9) No inactive management server node found 2025-06-29 04:13:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7b96d1d]) (logid:774460a9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e64e805]) (logid:c16de7f2) No inactive management server node found 2025-06-29 04:13:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e64e805]) (logid:c16de7f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:26,147 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-36d8e57b]) (logid:f70eaf0f) HA health check task is running... 2025-06-29 04:13:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4d86774]) (logid:c7507b31) No inactive management server node found 2025-06-29 04:13:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4d86774]) (logid:c7507b31) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-871a5cc7]) (logid:b6f0bb48) Enabled non-edge zones available for scan: 1 2025-06-29 04:13:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-871a5cc7]) (logid:b6f0bb48) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:13:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-871a5cc7]) (logid:b6f0bb48) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:13:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c8390259]) (logid:31ae2735) Enabled non-edge zones available for scan: 1 2025-06-29 04:13:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c8390259]) (logid:31ae2735) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:13:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f7a2450]) (logid:415463ff) No inactive management server node found 2025-06-29 04:13:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f7a2450]) (logid:415463ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c385187]) (logid:1a2518c8) No inactive management server node found 2025-06-29 04:13:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c385187]) (logid:1a2518c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:30,148 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bfc36139]) (logid:b5fe71fb) HA health check task is running... 2025-06-29 04:13:30,305 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8515: Seq 7-8515: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:30,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8515 2025-06-29 04:13:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0b9c194]) (logid:65bc4d91) No inactive management server node found 2025-06-29 04:13:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0b9c194]) (logid:65bc4d91) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-319d01cc]) (logid:dc164cd3) No inactive management server node found 2025-06-29 04:13:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-319d01cc]) (logid:dc164cd3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:34,150 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0c3a42fd]) (logid:7a8fc14f) HA health check task is running... 2025-06-29 04:13:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-323bb386]) (logid:bfced623) No inactive management server node found 2025-06-29 04:13:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-323bb386]) (logid:bfced623) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a56442]) (logid:4a317a84) No inactive management server node found 2025-06-29 04:13:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a56442]) (logid:4a317a84) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4c13a3c]) (logid:7dcfa04c) No inactive management server node found 2025-06-29 04:13:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4c13a3c]) (logid:7dcfa04c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:38,151 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8e42866d]) (logid:cb66dca5) HA health check task is running... 2025-06-29 04:13:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-745749e0]) (logid:679a3d24) No inactive management server node found 2025-06-29 04:13:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-745749e0]) (logid:679a3d24) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:40,304 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8516: Seq 7-8516: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:40,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8516 2025-06-29 04:13:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd1de7b4]) (logid:90e7262d) No inactive management server node found 2025-06-29 04:13:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd1de7b4]) (logid:90e7262d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-386f7a4d]) (logid:ac15cfbd) No inactive management server node found 2025-06-29 04:13:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-386f7a4d]) (logid:ac15cfbd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:42,152 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-7c91eae8]) (logid:6800ec70) HA health check task is running... 2025-06-29 04:13:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-99c523da]) (logid:379c017d) DbCollector is running... 2025-06-29 04:13:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a1d55f0b]) (logid:d90f3ade) Found 0 routers to update status. 2025-06-29 04:13:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a1d55f0b]) (logid:d90f3ade) Found 0 VPC's to update Redundant State. 2025-06-29 04:13:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a1d55f0b]) (logid:d90f3ade) Found 0 networks to update RvR status. 2025-06-29 04:13:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed801e18]) (logid:e2146878) No inactive management server node found 2025-06-29 04:13:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed801e18]) (logid:e2146878) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) ManagementServerCollector is running... 2025-06-29 04:13:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) DB properties were already loaded 2025-06-29 04:13:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:13:44,060 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Execution of process [4066383] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:13:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Exit value of process [4066383] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:13:44,061 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Process [4066383] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:13:44,061 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) usage install: null 2025-06-29 04:13:44,061 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Sessions active: 0 2025-06-29 04:13:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:13:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066386] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:13:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:13:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066391] for command [/bin/bash -c uname -r ]. 2025-06-29 04:13:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:13:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066392] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:13:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) system memory from /proc: 4887515136 2025-06-29 04:13:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:13:44,085 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066397] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:13:44,085 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) free memory from /proc: 299782144 2025-06-29 04:13:44,085 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:13:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066402] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:13:44,097 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) used memory from /proc: 1070980 2025-06-29 04:13:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:13:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066403] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:13:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:13:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066404] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:13:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:13:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066405] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:13:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:13:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066412] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:13:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:13:44,135 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Successfully executed process [4066413] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail buffersdirect.count, 93 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail buffersdirect.used, 321663 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail buffersmapped.capacity, 0 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail buffersmapped.count, 0 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail buffersmapped.used, 0 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail gcPS-MarkSweep.count, 5 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail gcPS-MarkSweep.time, 616 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail jvmuptime, 2283840 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memoryheap.committed, 500695040 2025-06-29 04:13:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memoryheap.init, 77594624 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memoryheap.usage, 0.2594569163543777 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorynon-heap.committed, 195493888 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorynon-heap.max, -1 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorynon-heap.usage, -1.93304096E8 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorynon-heap.used, 193304096 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11119743068515063 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21520186956376847 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.Compressed-Class-Space.usage, 0.01238223910331726 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.Metaspace.usage, 0.9936125169436586 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.PS-Eden-Space.usage, 0.4551541834319151 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.PS-Old-Gen.usage, 0.1306782362444503 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorypools.PS-Survivor-Space.usage, 0.9977760314941406 2025-06-29 04:13:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorytotal.committed, 696188928 2025-06-29 04:13:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorytotal.init, 85262336 2025-06-29 04:13:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorytotal.max, 1908932607 2025-06-29 04:13:44,139 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail memorytotal.used, 688821432 2025-06-29 04:13:44,140 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail threadsdeadlocks, [] 2025-06-29 04:13:44,140 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail threadsnew.count, 0 2025-06-29 04:13:44,141 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:13:44,143 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-f5b587a9]) (logid:4b8b1d8e) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:13:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":6138256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2283764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":687858376,"totalCommitted":696188928,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":495285768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":299782144,"systemMemoryUsed":1070980,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.57,0.61],"systemCyclesUsage":[66409973,53831446,4589484542],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:13:44,144 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 75, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:13:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":6138256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2283764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":687858376,"totalCommitted":696188928,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":495285768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":299782144,"systemMemoryUsed":1070980,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.57,0.61],"systemCyclesUsage":[66409973,53831446,4589484542],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:13:44,144 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:13:44,151 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-19:[ctx-96721e04]) (logid:99bd0236) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:13:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":6138256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2283764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":687858376,"totalCommitted":696188928,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":495285768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":299782144,"systemMemoryUsed":1070980,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.57,0.61],"systemCyclesUsage":[66409973,53831446,4589484542],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:13:44,151 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:13:44,151 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 75, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:13:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":500695040,"freeJvmMemoryBytes":6138256,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2283764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":687858376,"totalCommitted":696188928,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":495285768,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":299782144,"systemMemoryUsed":1070980,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.57,0.61],"systemCyclesUsage":[66409973,53831446,4589484542],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:13:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5bd079e]) (logid:04f0496c) No inactive management server node found 2025-06-29 04:13:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5bd079e]) (logid:04f0496c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:45,304 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8517: Seq 7-8517: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:45,313 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8517 2025-06-29 04:13:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:13:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:13:46,154 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f43bf4dc]) (logid:d64f2f65) HA health check task is running... 2025-06-29 04:13:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae22468b]) (logid:71545aa5) No inactive management server node found 2025-06-29 04:13:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae22468b]) (logid:71545aa5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:13:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:13:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:13:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:13:47,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:13:47,091 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:13:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:13:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:13:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e573e422]) (logid:e6abbc54) No inactive management server node found 2025-06-29 04:13:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e573e422]) (logid:e6abbc54) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-27a87731]) (logid:7c9dce05) VM scheduler.poll is being called at 2025-06-29T02:14:00+0000 2025-06-29 04:13:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-27a87731]) (logid:7c9dce05) Got 0 scheduled jobs to be executed at 2025-06-29T02:14:00+0000 2025-06-29 04:13:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-27a87731]) (logid:7c9dce05) Cleaned up 0 VM scheduled job entries 2025-06-29 04:13:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-c557e7b4]) (logid:34f1b0a6) VM Operation Thread Running 2025-06-29 04:13:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-390f1839]) (logid:d0b8ac3d) ClusterDRS.poll is being called at 2025-06-29T02:14:00+0000 2025-06-29 04:13:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-390f1839]) (logid:d0b8ac3d) Removed 0 old drs migration plans 2025-06-29 04:13:49,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91c30316]) (logid:78f489a5) No inactive management server node found 2025-06-29 04:13:49,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91c30316]) (logid:78f489a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:50,155 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-108206cf]) (logid:a82697b7) HA health check task is running... 2025-06-29 04:13:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54cd47c3]) (logid:8f4aa33b) No inactive management server node found 2025-06-29 04:13:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54cd47c3]) (logid:8f4aa33b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81c05543]) (logid:debc904f) No inactive management server node found 2025-06-29 04:13:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81c05543]) (logid:debc904f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b402b518]) (logid:7ae0fd0d) No inactive management server node found 2025-06-29 04:13:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b402b518]) (logid:7ae0fd0d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:54,156 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5618f2b6]) (logid:4c7a4c9c) HA health check task is running... 2025-06-29 04:13:55,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8519: Seq 7-8519: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:13:55,686 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:13:55,688 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:13:55,696 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9e44506]) (logid:8e424687) No inactive management server node found 2025-06-29 04:13:55,696 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c9e44506]) (logid:8e424687) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:55,696 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8519 2025-06-29 04:13:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83b62b01]) (logid:f3f82287) No inactive management server node found 2025-06-29 04:13:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83b62b01]) (logid:f3f82287) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-907fba8e]) (logid:3b3cf935) Enabled non-edge zones available for scan: 1 2025-06-29 04:13:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-907fba8e]) (logid:3b3cf935) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:13:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-907fba8e]) (logid:3b3cf935) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:13:58,158 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ed754aa7]) (logid:3136086d) HA health check task is running... 2025-06-29 04:13:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d991132b]) (logid:bedd2e6c) Enabled non-edge zones available for scan: 1 2025-06-29 04:13:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d991132b]) (logid:bedd2e6c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:13:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d30ff825]) (logid:433c8ec2) No inactive management server node found 2025-06-29 04:13:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d30ff825]) (logid:433c8ec2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:13:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dc367cd]) (logid:877d20ae) No inactive management server node found 2025-06-29 04:13:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dc367cd]) (logid:877d20ae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ec14a6d]) (logid:7fc462a8) No inactive management server node found 2025-06-29 04:14:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ec14a6d]) (logid:7fc462a8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:02,159 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-425edf85]) (logid:b3d5ab3f) HA health check task is running... 2025-06-29 04:14:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93324a8a]) (logid:f5c2edb7) No inactive management server node found 2025-06-29 04:14:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93324a8a]) (logid:f5c2edb7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb89798]) (logid:d3bb01e1) No inactive management server node found 2025-06-29 04:14:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bb89798]) (logid:d3bb01e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:05,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8520: Seq 7-8520: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:14:05,274 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8520 2025-06-29 04:14:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1743e198]) (logid:7af146dd) No inactive management server node found 2025-06-29 04:14:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1743e198]) (logid:7af146dd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:06,160 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-48b856eb]) (logid:d2888c82) HA health check task is running... 2025-06-29 04:14:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b7108c9]) (logid:9461d24b) No inactive management server node found 2025-06-29 04:14:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b7108c9]) (logid:9461d24b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3272e36c]) (logid:e7314509) No inactive management server node found 2025-06-29 04:14:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3272e36c]) (logid:e7314509) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:10,161 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ea83c730]) (logid:6df246b4) HA health check task is running... 2025-06-29 04:14:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83935e65]) (logid:a223d0e4) No inactive management server node found 2025-06-29 04:14:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83935e65]) (logid:a223d0e4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faf7b7a2]) (logid:f6f720ac) No inactive management server node found 2025-06-29 04:14:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faf7b7a2]) (logid:f6f720ac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a10f3e08]) (logid:1f40cc45) Found 0 routers to update status. 2025-06-29 04:14:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a10f3e08]) (logid:1f40cc45) Found 0 VPC's to update Redundant State. 2025-06-29 04:14:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a10f3e08]) (logid:1f40cc45) Found 0 networks to update RvR status. 2025-06-29 04:14:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d357c77]) (logid:415424b8) No inactive management server node found 2025-06-29 04:14:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d357c77]) (logid:415424b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:14,163 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-38cc9d2a]) (logid:3e990d64) HA health check task is running... 2025-06-29 04:14:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12872c01]) (logid:d6d3f7bb) No inactive management server node found 2025-06-29 04:14:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12872c01]) (logid:d6d3f7bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:15,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8521: Seq 7-8521: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:14:15,316 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8521 2025-06-29 04:14:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7afd4ec3]) (logid:a0ce243e) No inactive management server node found 2025-06-29 04:14:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7afd4ec3]) (logid:a0ce243e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71481a71]) (logid:f2de0194) No inactive management server node found 2025-06-29 04:14:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71481a71]) (logid:f2de0194) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:18,004 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-93a2df30]) (logid:4f8c9438) AutoScaling Monitor is running... 2025-06-29 04:14:18,066 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-254bf627]) (logid:9e8e1423) Starting to remove all volume_stats rows older than [Sat Jun 28 16:14:18 CEST 2025]. 2025-06-29 04:14:18,066 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-254bf627]) (logid:9e8e1423) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:14:18 CEST 2025]. 2025-06-29 04:14:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8c9e996d]) (logid:dffb7681) Resetting hosts suitable for reconnect 2025-06-29 04:14:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8c9e996d]) (logid:dffb7681) Completed resetting hosts suitable for reconnect 2025-06-29 04:14:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8c9e996d]) (logid:dffb7681) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:14:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8c9e996d]) (logid:dffb7681) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:14:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8c9e996d]) (logid:dffb7681) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:14:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8c9e996d]) (logid:dffb7681) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:14:18,164 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ab9dc290]) (logid:9cc722bb) HA health check task is running... 2025-06-29 04:14:18,297 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-ef9bbca5]) (logid:29b20e56) Starting to remove all vm_stats rows older than [Sat Jun 28 16:14:18 CEST 2025]. 2025-06-29 04:14:18,322 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-ef9bbca5]) (logid:29b20e56) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:14:18 CEST 2025]. 2025-06-29 04:14:18,788 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:14:18,790 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:14:18,791 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) Seq 2-4725683383995203748: Routed from 90520738888109 2025-06-29 04:14:18,834 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) Seq 1-4725683383995203748: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:14:18,836 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:14:18,837 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) Seq 2-3926857400090362029: Routed from 90520738888109 2025-06-29 04:14:18,881 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-365cd1d5]) (logid:f89d4a97) Seq 2-3926857400090362029: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:14:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-088655b5]) (logid:bcf09127) No inactive management server node found 2025-06-29 04:14:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-088655b5]) (logid:bcf09127) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:20,141 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:14:20,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:14:20,145 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) Seq 2-4725683383995203749: Routed from 90520738888109 2025-06-29 04:14:20,199 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) Seq 1-4725683383995203749: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:14:20,271 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:14:20,272 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) Seq 2-3926857400090362030: Routed from 90520738888109 2025-06-29 04:14:20,383 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-d051eb60]) (logid:81b48b3d) Seq 2-3926857400090362030: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:14:20,555 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) StorageCollector is running... 2025-06-29 04:14:20,560 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:14:20,560 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:14:20,560 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:14:20,561 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3938116399158788141: Routed from 90520738888109 2025-06-29 04:14:20,619 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 4-3938116399158788141: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:14:20,622 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:14:20,622 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:14:20,622 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:14:20,623 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3926857400090362031: Routed from 90520738888109 2025-06-29 04:14:20,634 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3926857400090362031: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:14:20,645 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:14:20,645 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:14:20,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:14:20,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3926857400090362032: Routed from 90520738888109 2025-06-29 04:14:20,738 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3926857400090362032: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:14:20,748 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:14:20,748 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:14:20,748 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:14:20,749 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-4725683383995203750: Routed from 90520738888109 2025-06-29 04:14:20,794 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 1-4725683383995203750: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:14:20,795 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:14:20,795 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:14:20,795 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:14:20,796 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3926857400090362033: Routed from 90520738888109 2025-06-29 04:14:20,840 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-bf3ab649]) (logid:c6ceafea) Seq 2-3926857400090362033: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:14:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea16cca2]) (logid:c54e4128) No inactive management server node found 2025-06-29 04:14:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea16cca2]) (logid:c54e4128) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:22,165 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-55123063]) (logid:7a91d565) HA health check task is running... 2025-06-29 04:14:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca7b5cd7]) (logid:305fd614) No inactive management server node found 2025-06-29 04:14:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca7b5cd7]) (logid:305fd614) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a49da27d]) (logid:97f4e7a8) No inactive management server node found 2025-06-29 04:14:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a49da27d]) (logid:97f4e7a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:25,307 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8522: Seq 7-8522: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:14:25,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8522 2025-06-29 04:14:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54424075]) (logid:3d414fc1) No inactive management server node found 2025-06-29 04:14:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54424075]) (logid:3d414fc1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:26,166 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-fe72460e]) (logid:c076dc05) HA health check task is running... 2025-06-29 04:14:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61ad5b6c]) (logid:5e6765a6) No inactive management server node found 2025-06-29 04:14:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61ad5b6c]) (logid:5e6765a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-84888b64]) (logid:2b64b049) Enabled non-edge zones available for scan: 1 2025-06-29 04:14:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-84888b64]) (logid:2b64b049) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:14:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-84888b64]) (logid:2b64b049) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:14:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-874ce393]) (logid:6632c2a9) Enabled non-edge zones available for scan: 1 2025-06-29 04:14:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-874ce393]) (logid:6632c2a9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:14:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ea98940]) (logid:69ff0301) No inactive management server node found 2025-06-29 04:14:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ea98940]) (logid:69ff0301) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c05ef86]) (logid:7ce40a3b) No inactive management server node found 2025-06-29 04:14:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c05ef86]) (logid:7ce40a3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:30,168 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-52482fcf]) (logid:15aa857e) HA health check task is running... 2025-06-29 04:14:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79feb6ee]) (logid:5488ad3a) No inactive management server node found 2025-06-29 04:14:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79feb6ee]) (logid:5488ad3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30914870]) (logid:66bfd2e5) No inactive management server node found 2025-06-29 04:14:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30914870]) (logid:66bfd2e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:34,169 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6ba2b874]) (logid:cc2e9e5b) HA health check task is running... 2025-06-29 04:14:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b0889fb]) (logid:6415a3d3) No inactive management server node found 2025-06-29 04:14:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b0889fb]) (logid:6415a3d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:35,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8523: Seq 7-8523: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:14:35,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8523 2025-06-29 04:14:35,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b813446b]) (logid:4b3f3d91) No inactive management server node found 2025-06-29 04:14:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b813446b]) (logid:4b3f3d91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3726ac1]) (logid:7ef03eda) No inactive management server node found 2025-06-29 04:14:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f3726ac1]) (logid:7ef03eda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:38,170 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ef01dd9d]) (logid:20863de1) HA health check task is running... 2025-06-29 04:14:38,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-431e8907]) (logid:619f55ad) No inactive management server node found 2025-06-29 04:14:38,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-431e8907]) (logid:619f55ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a14b34]) (logid:6b3e801b) No inactive management server node found 2025-06-29 04:14:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a14b34]) (logid:6b3e801b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24a558d9]) (logid:cae5147f) No inactive management server node found 2025-06-29 04:14:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-24a558d9]) (logid:cae5147f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:42,171 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-58a5e0b1]) (logid:87ee9bec) HA health check task is running... 2025-06-29 04:14:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-96c55d1c]) (logid:7bac287f) DbCollector is running... 2025-06-29 04:14:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5ccec47]) (logid:8cf16426) Found 0 routers to update status. 2025-06-29 04:14:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5ccec47]) (logid:8cf16426) Found 0 VPC's to update Redundant State. 2025-06-29 04:14:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5ccec47]) (logid:8cf16426) Found 0 networks to update RvR status. 2025-06-29 04:14:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-674dbdaa]) (logid:b3242b11) No inactive management server node found 2025-06-29 04:14:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-674dbdaa]) (logid:b3242b11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) ManagementServerCollector is running... 2025-06-29 04:14:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) DB properties were already loaded 2025-06-29 04:14:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:14:44,056 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Execution of process [4066427] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:14:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Exit value of process [4066427] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:14:44,057 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Process [4066427] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:14:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) usage install: null 2025-06-29 04:14:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Sessions active: 0 2025-06-29 04:14:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:14:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066430] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:14:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:14:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066435] for command [/bin/bash -c uname -r ]. 2025-06-29 04:14:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:14:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066436] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:14:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) system memory from /proc: 4887515136 2025-06-29 04:14:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:14:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066441] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:14:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) free memory from /proc: 302067712 2025-06-29 04:14:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:14:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066446] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:14:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) used memory from /proc: 1068540 2025-06-29 04:14:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:14:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066447] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:14:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:14:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066448] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:14:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:14:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066449] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:14:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:14:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066456] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:14:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:14:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Successfully executed process [4066457] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:14:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:14:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail buffersdirect.count, 93 2025-06-29 04:14:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail buffersdirect.used, 321663 2025-06-29 04:14:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail buffersmapped.capacity, 0 2025-06-29 04:14:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail buffersmapped.count, 0 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail buffersmapped.used, 0 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail jvmuptime, 2343833 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memoryheap.committed, 620232704 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memoryheap.init, 77594624 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memoryheap.usage, 0.10678072507418764 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorynon-heap.committed, 195756032 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorynon-heap.max, -1 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorynon-heap.usage, -1.9358884E8 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorynon-heap.used, 193588840 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11244084910690483 2025-06-29 04:14:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21621621621621623 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.Compressed-Class-Space.usage, 0.01238223910331726 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.Metaspace.usage, 0.9936646025791195 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.PS-Eden-Space.usage, 0.035232434780684864 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorytotal.committed, 815988736 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorytotal.init, 85262336 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorytotal.max, 1908932607 2025-06-29 04:14:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail memorytotal.used, 397651456 2025-06-29 04:14:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail threadsdeadlocks, [] 2025-06-29 04:14:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail threadsnew.count, 0 2025-06-29 04:14:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:14:44,136 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-ab2029b9]) (logid:8eb13a5a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:14:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":417142416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2343760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":396669096,"totalCommitted":815988736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203837208,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302067712,"systemMemoryUsed":1068540,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.52,0.59],"systemCyclesUsage":[66410508,53831766,4589507440],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:14:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 77, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:14:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":417142416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2343760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":396669096,"totalCommitted":815988736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203837208,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302067712,"systemMemoryUsed":1068540,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.52,0.59],"systemCyclesUsage":[66410508,53831766,4589507440],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:14:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:14:44,142 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-19:[ctx-ec4355a1]) (logid:d537e60a) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:14:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":417142416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2343760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":396669096,"totalCommitted":815988736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203837208,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302067712,"systemMemoryUsed":1068540,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.52,0.59],"systemCyclesUsage":[66410508,53831766,4589507440],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:14:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:14:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 77, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:14:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":417142416,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2343760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.19,"totalInit":85262336,"totalUsed":396669096,"totalCommitted":815988736,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":203837208,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302067712,"systemMemoryUsed":1068540,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.19,0.52,0.59],"systemCyclesUsage":[66410508,53831766,4589507440],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:14:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66bbb3d4]) (logid:cc8de058) No inactive management server node found 2025-06-29 04:14:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66bbb3d4]) (logid:cc8de058) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:45,307 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8524: Seq 7-8524: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:14:45,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8524 2025-06-29 04:14:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:14:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:14:46,173 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e80a908b]) (logid:22c80bc8) HA health check task is running... 2025-06-29 04:14:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d646b800]) (logid:8a8f3258) No inactive management server node found 2025-06-29 04:14:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d646b800]) (logid:8a8f3258) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:14:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:14:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:14:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:14:47,069 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:14:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:14:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:14:47,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:14:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cf7d108]) (logid:5e7fa2ff) No inactive management server node found 2025-06-29 04:14:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cf7d108]) (logid:5e7fa2ff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b7246270]) (logid:b094bb06) VM scheduler.poll is being called at 2025-06-29T02:15:00+0000 2025-06-29 04:14:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b7246270]) (logid:b094bb06) Got 0 scheduled jobs to be executed at 2025-06-29T02:15:00+0000 2025-06-29 04:14:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b7246270]) (logid:b094bb06) Cleaned up 0 VM scheduled job entries 2025-06-29 04:14:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-4f5edb19]) (logid:5ff7a31c) VM Operation Thread Running 2025-06-29 04:14:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-999444fe]) (logid:b057eb62) ClusterDRS.poll is being called at 2025-06-29T02:15:00+0000 2025-06-29 04:14:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-999444fe]) (logid:b057eb62) Removed 0 old drs migration plans 2025-06-29 04:14:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aebcb65]) (logid:92016aff) No inactive management server node found 2025-06-29 04:14:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aebcb65]) (logid:92016aff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:50,174 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-670bb5cb]) (logid:2284239d) HA health check task is running... 2025-06-29 04:14:50,307 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8526: Seq 7-8526: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:14:50,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8526 2025-06-29 04:14:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68d7b103]) (logid:b33adc1f) No inactive management server node found 2025-06-29 04:14:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68d7b103]) (logid:b33adc1f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:14:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73c26ff5]) (logid:47f24536) No inactive management server node found 2025-06-29 04:14:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73c26ff5]) (logid:47f24536) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea02a6f4]) (logid:311e73bc) No inactive management server node found 2025-06-29 04:14:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea02a6f4]) (logid:311e73bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:54,175 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-74977970]) (logid:03cc3345) HA health check task is running... 2025-06-29 04:14:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f4b9f00]) (logid:f6face64) No inactive management server node found 2025-06-29 04:14:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f4b9f00]) (logid:f6face64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a1d3b2b]) (logid:cbdf3742) No inactive management server node found 2025-06-29 04:14:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a1d3b2b]) (logid:cbdf3742) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4394400e]) (logid:742d4c15) Enabled non-edge zones available for scan: 1 2025-06-29 04:14:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4394400e]) (logid:742d4c15) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:14:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4394400e]) (logid:742d4c15) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:14:58,176 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a74960e3]) (logid:cd652ab7) HA health check task is running... 2025-06-29 04:14:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3ad367c9]) (logid:1b84dc3b) Enabled non-edge zones available for scan: 1 2025-06-29 04:14:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3ad367c9]) (logid:1b84dc3b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:14:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-582602e3]) (logid:cbea28ea) No inactive management server node found 2025-06-29 04:14:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-582602e3]) (logid:cbea28ea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:14:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b00385c]) (logid:3269c210) No inactive management server node found 2025-06-29 04:14:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b00385c]) (logid:3269c210) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:00,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8527: Seq 7-8527: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:15:00,319 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8527 2025-06-29 04:15:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2a2161f]) (logid:e1bf9fef) No inactive management server node found 2025-06-29 04:15:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2a2161f]) (logid:e1bf9fef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:02,177 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-45096640]) (logid:59b292f2) HA health check task is running... 2025-06-29 04:15:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b798371]) (logid:d748a6a4) No inactive management server node found 2025-06-29 04:15:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b798371]) (logid:d748a6a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab63f14c]) (logid:6601addb) No inactive management server node found 2025-06-29 04:15:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab63f14c]) (logid:6601addb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29a70737]) (logid:67466a8f) No inactive management server node found 2025-06-29 04:15:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29a70737]) (logid:67466a8f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:06,179 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-900ab090]) (logid:af29cc99) HA health check task is running... 2025-06-29 04:15:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-091305d5]) (logid:2c800dae) No inactive management server node found 2025-06-29 04:15:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-091305d5]) (logid:2c800dae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbc37821]) (logid:524003af) No inactive management server node found 2025-06-29 04:15:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbc37821]) (logid:524003af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:10,180 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a1ea0fc2]) (logid:9f3ae81e) HA health check task is running... 2025-06-29 04:15:10,308 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8528: Seq 7-8528: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:15:10,318 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8528 2025-06-29 04:15:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9239edd7]) (logid:c6bc4ba5) No inactive management server node found 2025-06-29 04:15:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9239edd7]) (logid:c6bc4ba5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73c9b621]) (logid:81cdeccd) No inactive management server node found 2025-06-29 04:15:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73c9b621]) (logid:81cdeccd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8276d8b4]) (logid:c22b4288) Found 0 routers to update status. 2025-06-29 04:15:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8276d8b4]) (logid:c22b4288) Found 0 VPC's to update Redundant State. 2025-06-29 04:15:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8276d8b4]) (logid:c22b4288) Found 0 networks to update RvR status. 2025-06-29 04:15:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e365b81a]) (logid:8d3bf976) No inactive management server node found 2025-06-29 04:15:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e365b81a]) (logid:8d3bf976) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:14,182 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-608a3d72]) (logid:79e44f2b) HA health check task is running... 2025-06-29 04:15:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e17ddc75]) (logid:c57d8cf1) No inactive management server node found 2025-06-29 04:15:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e17ddc75]) (logid:c57d8cf1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-516fcefd]) (logid:1c108d5a) No inactive management server node found 2025-06-29 04:15:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-516fcefd]) (logid:1c108d5a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de4da42b]) (logid:2f03e430) No inactive management server node found 2025-06-29 04:15:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de4da42b]) (logid:2f03e430) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:18,006 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-b34abeb6]) (logid:97954afc) AutoScaling Monitor is running... 2025-06-29 04:15:18,068 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-e148bd9f]) (logid:326ebf1f) Starting to remove all volume_stats rows older than [Sat Jun 28 16:15:18 CEST 2025]. 2025-06-29 04:15:18,069 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-e148bd9f]) (logid:326ebf1f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:15:18 CEST 2025]. 2025-06-29 04:15:18,183 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e6b452f7]) (logid:bc16bc89) HA health check task is running... 2025-06-29 04:15:18,324 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-58184e16]) (logid:da3b29b4) Starting to remove all vm_stats rows older than [Sat Jun 28 16:15:18 CEST 2025]. 2025-06-29 04:15:18,348 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-58184e16]) (logid:da3b29b4) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:15:18 CEST 2025]. 2025-06-29 04:15:18,883 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:15:18,884 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:15:18,886 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) Seq 2-4725683383995203751: Routed from 90520738888109 2025-06-29 04:15:18,930 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) Seq 1-4725683383995203751: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:15:18,931 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:15:18,931 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) Seq 2-3926857400090362034: Routed from 90520738888109 2025-06-29 04:15:18,976 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-661558fb]) (logid:553fb0fa) Seq 2-3926857400090362034: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:15:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb8a51cd]) (logid:06b8bf9c) No inactive management server node found 2025-06-29 04:15:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb8a51cd]) (logid:06b8bf9c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:20,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8529: Seq 7-8529: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:15:20,319 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8529 2025-06-29 04:15:20,447 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:15:20,450 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:15:20,451 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) Seq 2-4725683383995203752: Routed from 90520738888109 2025-06-29 04:15:20,505 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) Seq 1-4725683383995203752: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:15:20,541 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:15:20,542 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) Seq 2-3926857400090362035: Routed from 90520738888109 2025-06-29 04:15:20,629 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-f40fb4a2]) (logid:76b77361) Seq 2-3926857400090362035: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:15:20,841 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) StorageCollector is running... 2025-06-29 04:15:20,846 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:15:20,846 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:15:20,846 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:15:20,847 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3938116399158788142: Routed from 90520738888109 2025-06-29 04:15:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-119c5060]) (logid:4c9c7162) No inactive management server node found 2025-06-29 04:15:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-119c5060]) (logid:4c9c7162) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:20,901 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 4-3938116399158788142: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:15:20,904 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:15:20,904 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:15:20,904 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:15:20,905 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3926857400090362036: Routed from 90520738888109 2025-06-29 04:15:20,916 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3926857400090362036: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:15:20,929 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:15:20,929 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:15:20,929 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:15:20,930 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3926857400090362037: Routed from 90520738888109 2025-06-29 04:15:21,022 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3926857400090362037: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:15:21,039 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:15:21,039 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:15:21,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:15:21,040 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-4725683383995203753: Routed from 90520738888109 2025-06-29 04:15:21,083 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 1-4725683383995203753: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:15:21,085 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:15:21,085 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:15:21,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:15:21,086 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3926857400090362038: Routed from 90520738888109 2025-06-29 04:15:21,131 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-008fd849]) (logid:bb4a03cc) Seq 2-3926857400090362038: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:15:22,184 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-783df06d]) (logid:21cab874) HA health check task is running... 2025-06-29 04:15:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8414b98]) (logid:73328657) No inactive management server node found 2025-06-29 04:15:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8414b98]) (logid:73328657) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c9c1775]) (logid:fa03852b) No inactive management server node found 2025-06-29 04:15:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c9c1775]) (logid:fa03852b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e87f834]) (logid:1651dd68) No inactive management server node found 2025-06-29 04:15:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e87f834]) (logid:1651dd68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:26,185 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e134f330]) (logid:62e9aedb) HA health check task is running... 2025-06-29 04:15:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3690f2f]) (logid:dece6212) No inactive management server node found 2025-06-29 04:15:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3690f2f]) (logid:dece6212) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0acf9951]) (logid:dc6f9a6c) Enabled non-edge zones available for scan: 1 2025-06-29 04:15:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0acf9951]) (logid:dc6f9a6c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:15:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0acf9951]) (logid:dc6f9a6c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:15:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-05e41544]) (logid:b116299a) Enabled non-edge zones available for scan: 1 2025-06-29 04:15:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-05e41544]) (logid:b116299a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:15:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-753115da]) (logid:8c906fb5) No inactive management server node found 2025-06-29 04:15:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-753115da]) (logid:8c906fb5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31b3cddd]) (logid:e909b86f) No inactive management server node found 2025-06-29 04:15:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31b3cddd]) (logid:e909b86f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:30,186 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-af5d205d]) (logid:3c8133e1) HA health check task is running... 2025-06-29 04:15:30,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8530: Seq 7-8530: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:15:30,319 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8530 2025-06-29 04:15:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cbcf55b]) (logid:3d8c3ae9) No inactive management server node found 2025-06-29 04:15:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cbcf55b]) (logid:3d8c3ae9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91af9402]) (logid:7e386bd3) No inactive management server node found 2025-06-29 04:15:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91af9402]) (logid:7e386bd3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:34,188 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f151fadf]) (logid:fe301ecf) HA health check task is running... 2025-06-29 04:15:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bebb418]) (logid:634ce2cc) No inactive management server node found 2025-06-29 04:15:34,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bebb418]) (logid:634ce2cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:35,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cb2f7d0]) (logid:2e577f36) No inactive management server node found 2025-06-29 04:15:35,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cb2f7d0]) (logid:2e577f36) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-762dbbc1]) (logid:05eb6c26) No inactive management server node found 2025-06-29 04:15:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-762dbbc1]) (logid:05eb6c26) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:38,189 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ce552a98]) (logid:e3f9cca5) HA health check task is running... 2025-06-29 04:15:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00ef5364]) (logid:13280c59) No inactive management server node found 2025-06-29 04:15:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00ef5364]) (logid:13280c59) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:40,309 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8531: Seq 7-8531: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:15:40,320 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8531 2025-06-29 04:15:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41fd394a]) (logid:0f8ed719) No inactive management server node found 2025-06-29 04:15:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-41fd394a]) (logid:0f8ed719) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18f19cef]) (logid:c94f903d) No inactive management server node found 2025-06-29 04:15:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18f19cef]) (logid:c94f903d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:42,190 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d2632549]) (logid:3179505f) HA health check task is running... 2025-06-29 04:15:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-dbf5da5d]) (logid:34dc6a8d) DbCollector is running... 2025-06-29 04:15:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a1a28c40]) (logid:dc20fece) Found 0 routers to update status. 2025-06-29 04:15:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a1a28c40]) (logid:dc20fece) Found 0 VPC's to update Redundant State. 2025-06-29 04:15:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a1a28c40]) (logid:dc20fece) Found 0 networks to update RvR status. 2025-06-29 04:15:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8044ff04]) (logid:f2a1f0cc) No inactive management server node found 2025-06-29 04:15:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8044ff04]) (logid:f2a1f0cc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:44,042 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) ManagementServerCollector is running... 2025-06-29 04:15:44,044 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) DB properties were already loaded 2025-06-29 04:15:44,044 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:15:44,059 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Execution of process [4066462] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:15:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Exit value of process [4066462] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:15:44,060 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Process [4066462] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:15:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) usage install: null 2025-06-29 04:15:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Sessions active: 0 2025-06-29 04:15:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:15:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066465] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:15:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:15:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066470] for command [/bin/bash -c uname -r ]. 2025-06-29 04:15:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:15:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066471] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:15:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) system memory from /proc: 4887515136 2025-06-29 04:15:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:15:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066476] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:15:44,084 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) free memory from /proc: 293310464 2025-06-29 04:15:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:15:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066481] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:15:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) used memory from /proc: 1070588 2025-06-29 04:15:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:15:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066482] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:15:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:15:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066483] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:15:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:15:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066484] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:15:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:15:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066491] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:15:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:15:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Successfully executed process [4066492] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail buffersdirect.count, 93 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail buffersdirect.used, 321663 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail buffersmapped.capacity, 0 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail buffersmapped.count, 0 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail buffersmapped.used, 0 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:15:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail jvmuptime, 2403834 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memoryheap.committed, 620232704 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memoryheap.init, 77594624 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memoryheap.usage, 0.1195964525113293 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorynon-heap.committed, 195952640 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorynon-heap.max, -1 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorynon-heap.usage, -1.93672056E8 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorynon-heap.used, 193672056 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11258664356171688 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21666194554603926 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.Compressed-Class-Space.usage, 0.01238223910331726 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.Metaspace.usage, 0.9932719238741459 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.PS-Eden-Space.usage, 0.0723246951929526 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorytotal.committed, 816185344 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorytotal.init, 85262336 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorytotal.max, 1908932607 2025-06-29 04:15:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail memorytotal.used, 421973624 2025-06-29 04:15:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail threadsdeadlocks, [] 2025-06-29 04:15:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail threadsnew.count, 0 2025-06-29 04:15:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:15:44,134 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-d54243e7]) (logid:8a759001) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:15:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":392744848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2403763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.24,"totalInit":85262336,"totalUsed":421154536,"totalCommitted":816185344,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228301568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":293310464,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.24,0.47,0.57],"systemCyclesUsage":[66410828,53832071,4589530568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:15:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 79, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:15:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":392744848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2403763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.24,"totalInit":85262336,"totalUsed":421154536,"totalCommitted":816185344,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228301568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":293310464,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.24,0.47,0.57],"systemCyclesUsage":[66410828,53832071,4589530568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:15:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:15:44,141 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-19:[ctx-3a9c7888]) (logid:b9a1aade) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:15:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":392744848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2403763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.24,"totalInit":85262336,"totalUsed":421154536,"totalCommitted":816185344,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228301568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":293310464,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.24,0.47,0.57],"systemCyclesUsage":[66410828,53832071,4589530568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:15:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:15:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 79, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:15:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":392744848,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2403763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.24,"totalInit":85262336,"totalUsed":421154536,"totalCommitted":816185344,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":228301568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":293310464,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.24,0.47,0.57],"systemCyclesUsage":[66410828,53832071,4589530568],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:15:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d694bfab]) (logid:75458a98) No inactive management server node found 2025-06-29 04:15:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d694bfab]) (logid:75458a98) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:15:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:15:46,192 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-911ecdbf]) (logid:6da6bdcd) HA health check task is running... 2025-06-29 04:15:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04fb39a7]) (logid:70a86c88) No inactive management server node found 2025-06-29 04:15:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04fb39a7]) (logid:70a86c88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:15:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:15:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:15:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:15:47,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:15:47,121 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:15:47,122 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:15:47,124 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:15:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be715109]) (logid:401bff66) No inactive management server node found 2025-06-29 04:15:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be715109]) (logid:401bff66) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:48,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-97c55251]) (logid:13fe5f01) Resetting hosts suitable for reconnect 2025-06-29 04:15:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-97c55251]) (logid:13fe5f01) Completed resetting hosts suitable for reconnect 2025-06-29 04:15:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-97c55251]) (logid:13fe5f01) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:15:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-97c55251]) (logid:13fe5f01) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:15:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-97c55251]) (logid:13fe5f01) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:15:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-97c55251]) (logid:13fe5f01) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:15:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-57c24b6f]) (logid:60b53310) VM scheduler.poll is being called at 2025-06-29T02:16:00+0000 2025-06-29 04:15:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-57c24b6f]) (logid:60b53310) Got 0 scheduled jobs to be executed at 2025-06-29T02:16:00+0000 2025-06-29 04:15:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-57c24b6f]) (logid:60b53310) Cleaned up 0 VM scheduled job entries 2025-06-29 04:15:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-6d95e834]) (logid:599604f2) VM Operation Thread Running 2025-06-29 04:15:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-755d967c]) (logid:bfab618c) ClusterDRS.poll is being called at 2025-06-29T02:16:00+0000 2025-06-29 04:15:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-755d967c]) (logid:bfab618c) Removed 0 old drs migration plans 2025-06-29 04:15:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321a0cff]) (logid:3621a77c) No inactive management server node found 2025-06-29 04:15:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321a0cff]) (logid:3621a77c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:50,193 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1bfeb66f]) (logid:d6355747) HA health check task is running... 2025-06-29 04:15:50,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8533: Seq 7-8533: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:15:50,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8533 2025-06-29 04:15:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0b7f6a2]) (logid:79f8ce02) No inactive management server node found 2025-06-29 04:15:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0b7f6a2]) (logid:79f8ce02) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f268cff2]) (logid:6feb3509) No inactive management server node found 2025-06-29 04:15:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f268cff2]) (logid:6feb3509) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:15:53,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f833a64]) (logid:804dd139) No inactive management server node found 2025-06-29 04:15:53,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f833a64]) (logid:804dd139) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:54,194 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-eed49a6b]) (logid:d493dff7) HA health check task is running... 2025-06-29 04:15:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14eca421]) (logid:724b0a50) No inactive management server node found 2025-06-29 04:15:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14eca421]) (logid:724b0a50) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7adf5ee]) (logid:d7bdc1b8) No inactive management server node found 2025-06-29 04:15:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7adf5ee]) (logid:d7bdc1b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-29a32c5e]) (logid:3e6b357d) Enabled non-edge zones available for scan: 1 2025-06-29 04:15:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-29a32c5e]) (logid:3e6b357d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:15:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-29a32c5e]) (logid:3e6b357d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:15:58,195 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dc6d1e5a]) (logid:4db18ecc) HA health check task is running... 2025-06-29 04:15:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-68b3d657]) (logid:76b170e6) Enabled non-edge zones available for scan: 1 2025-06-29 04:15:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-68b3d657]) (logid:76b170e6) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:15:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df408f47]) (logid:72f3d811) No inactive management server node found 2025-06-29 04:15:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df408f47]) (logid:72f3d811) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:15:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43be3511]) (logid:c6a62597) No inactive management server node found 2025-06-29 04:15:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43be3511]) (logid:c6a62597) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:00,310 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8534: Seq 7-8534: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:00,320 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8534 2025-06-29 04:16:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6d48444]) (logid:03cce875) No inactive management server node found 2025-06-29 04:16:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6d48444]) (logid:03cce875) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:02,197 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-cf97f7ba]) (logid:f96dfe10) HA health check task is running... 2025-06-29 04:16:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4e7623e]) (logid:45263f97) No inactive management server node found 2025-06-29 04:16:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4e7623e]) (logid:45263f97) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-351cc3b4]) (logid:1d151f32) No inactive management server node found 2025-06-29 04:16:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-351cc3b4]) (logid:1d151f32) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f52ecfb6]) (logid:d7ea9bd4) No inactive management server node found 2025-06-29 04:16:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f52ecfb6]) (logid:d7ea9bd4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:06,198 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-336a57ef]) (logid:a47e50f8) HA health check task is running... 2025-06-29 04:16:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80a7443d]) (logid:8dc7edd7) No inactive management server node found 2025-06-29 04:16:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80a7443d]) (logid:8dc7edd7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13b61a11]) (logid:f5631894) No inactive management server node found 2025-06-29 04:16:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13b61a11]) (logid:f5631894) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:10,200 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-254fa706]) (logid:ce691fa6) HA health check task is running... 2025-06-29 04:16:10,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8535: Seq 7-8535: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:10,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8535 2025-06-29 04:16:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed94b9cc]) (logid:b9b71715) No inactive management server node found 2025-06-29 04:16:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed94b9cc]) (logid:b9b71715) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6896324]) (logid:da1afff4) No inactive management server node found 2025-06-29 04:16:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6896324]) (logid:da1afff4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f280e226]) (logid:d20b76a0) Found 0 routers to update status. 2025-06-29 04:16:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f280e226]) (logid:d20b76a0) Found 0 VPC's to update Redundant State. 2025-06-29 04:16:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f280e226]) (logid:d20b76a0) Found 0 networks to update RvR status. 2025-06-29 04:16:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-712299b1]) (logid:a4b0e804) No inactive management server node found 2025-06-29 04:16:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-712299b1]) (logid:a4b0e804) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:14,201 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-19fb24cf]) (logid:a611ec89) HA health check task is running... 2025-06-29 04:16:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87fb840e]) (logid:b0e7b3fd) No inactive management server node found 2025-06-29 04:16:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87fb840e]) (logid:b0e7b3fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c0e4d5e]) (logid:7cc1ba7d) No inactive management server node found 2025-06-29 04:16:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c0e4d5e]) (logid:7cc1ba7d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29ba4cff]) (logid:ffe805e6) No inactive management server node found 2025-06-29 04:16:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29ba4cff]) (logid:ffe805e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:18,008 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-412c6e8d]) (logid:8eb91ce6) AutoScaling Monitor is running... 2025-06-29 04:16:18,071 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-f6cdccf6]) (logid:ac3560c3) Starting to remove all volume_stats rows older than [Sat Jun 28 16:16:18 CEST 2025]. 2025-06-29 04:16:18,071 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-f6cdccf6]) (logid:ac3560c3) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:16:18 CEST 2025]. 2025-06-29 04:16:18,202 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d9ebc6c3]) (logid:4e8ef15e) HA health check task is running... 2025-06-29 04:16:18,349 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-aef9a097]) (logid:27384053) Starting to remove all vm_stats rows older than [Sat Jun 28 16:16:18 CEST 2025]. 2025-06-29 04:16:18,371 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-aef9a097]) (logid:27384053) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:16:18 CEST 2025]. 2025-06-29 04:16:18,979 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:16:18,981 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:16:18,983 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) Seq 2-4725683383995203754: Routed from 90520738888109 2025-06-29 04:16:19,027 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) Seq 1-4725683383995203754: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:16:19,029 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:16:19,030 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) Seq 2-3926857400090362039: Routed from 90520738888109 2025-06-29 04:16:19,074 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-291bcd57]) (logid:c3ff262e) Seq 2-3926857400090362039: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:16:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d37b0020]) (logid:d22e8622) No inactive management server node found 2025-06-29 04:16:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d37b0020]) (logid:d22e8622) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:20,312 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8536: Seq 7-8536: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:20,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8536 2025-06-29 04:16:20,684 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:16:20,687 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:16:20,688 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) Seq 2-4725683383995203755: Routed from 90520738888109 2025-06-29 04:16:20,742 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) Seq 1-4725683383995203755: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:16:20,777 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:16:20,778 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) Seq 2-3926857400090362040: Routed from 90520738888109 2025-06-29 04:16:20,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75855d26]) (logid:71fa2e21) No inactive management server node found 2025-06-29 04:16:20,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75855d26]) (logid:71fa2e21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:20,876 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-1e3edbc0]) (logid:bd7e9402) Seq 2-3926857400090362040: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:16:21,131 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) StorageCollector is running... 2025-06-29 04:16:21,137 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:16:21,137 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:16:21,137 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:16:21,138 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-3938116399158788143: Routed from 90520738888109 2025-06-29 04:16:21,199 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 4-3938116399158788143: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:16:21,203 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:16:21,203 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:16:21,203 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:16:21,204 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-3926857400090362041: Routed from 90520738888109 2025-06-29 04:16:21,217 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-3926857400090362041: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:16:21,229 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:16:21,229 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:16:21,229 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:16:21,230 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-4725683383995203756: Routed from 90520738888109 2025-06-29 04:16:21,277 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 1-4725683383995203756: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:16:21,287 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:16:21,287 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:16:21,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:16:21,288 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-4725683383995203757: Routed from 90520738888109 2025-06-29 04:16:21,374 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 1-4725683383995203757: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:16:21,376 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:16:21,376 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:16:21,376 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:16:21,377 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-3926857400090362042: Routed from 90520738888109 2025-06-29 04:16:21,422 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5e8b91fe]) (logid:1e0127b2) Seq 2-3926857400090362042: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:16:22,203 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c955a644]) (logid:72e3312d) HA health check task is running... 2025-06-29 04:16:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4942c7b7]) (logid:8f0ca314) No inactive management server node found 2025-06-29 04:16:22,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4942c7b7]) (logid:8f0ca314) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:16:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b60b968e]) (logid:3226fe54) No inactive management server node found 2025-06-29 04:16:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b60b968e]) (logid:3226fe54) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:25,289 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 2-3926857400090361857: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10eb1c9b]) (logid:73136ca1) No inactive management server node found 2025-06-29 04:16:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10eb1c9b]) (logid:73136ca1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:26,205 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b01f7302]) (logid:bc4b3896) HA health check task is running... 2025-06-29 04:16:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89ae8550]) (logid:5e445065) No inactive management server node found 2025-06-29 04:16:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89ae8550]) (logid:5e445065) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ae53240a]) (logid:f4921f72) Enabled non-edge zones available for scan: 1 2025-06-29 04:16:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ae53240a]) (logid:f4921f72) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:16:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ae53240a]) (logid:f4921f72) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:16:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-581d6151]) (logid:1df0abc2) Enabled non-edge zones available for scan: 1 2025-06-29 04:16:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-581d6151]) (logid:1df0abc2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:16:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72dc52fc]) (logid:c3804020) No inactive management server node found 2025-06-29 04:16:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72dc52fc]) (logid:c3804020) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:29,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-900505cd]) (logid:0aa57f86) No inactive management server node found 2025-06-29 04:16:29,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-900505cd]) (logid:0aa57f86) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:30,206 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c051533e]) (logid:0c4392c9) HA health check task is running... 2025-06-29 04:16:30,311 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8537: Seq 7-8537: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:30,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8537 2025-06-29 04:16:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7658664]) (logid:cc7b2d11) No inactive management server node found 2025-06-29 04:16:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7658664]) (logid:cc7b2d11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d7feb44]) (logid:99596dad) No inactive management server node found 2025-06-29 04:16:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d7feb44]) (logid:99596dad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:34,208 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-71ea1a01]) (logid:afb5b41b) HA health check task is running... 2025-06-29 04:16:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b88267a7]) (logid:f88bfac4) No inactive management server node found 2025-06-29 04:16:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b88267a7]) (logid:f88bfac4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b500cb6]) (logid:14714d24) No inactive management server node found 2025-06-29 04:16:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b500cb6]) (logid:14714d24) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1da6dc0]) (logid:165843f7) No inactive management server node found 2025-06-29 04:16:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1da6dc0]) (logid:165843f7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:38,209 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a671e520]) (logid:9f02d930) HA health check task is running... 2025-06-29 04:16:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c04fb0bc]) (logid:7180ab18) No inactive management server node found 2025-06-29 04:16:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c04fb0bc]) (logid:7180ab18) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:40,313 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8538: Seq 7-8538: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:40,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8538 2025-06-29 04:16:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6344ccca]) (logid:b0a466a6) No inactive management server node found 2025-06-29 04:16:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6344ccca]) (logid:b0a466a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a40cefb4]) (logid:86d4d479) No inactive management server node found 2025-06-29 04:16:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a40cefb4]) (logid:86d4d479) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:42,210 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a824b669]) (logid:d606e3fd) HA health check task is running... 2025-06-29 04:16:43,014 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-a8b6d53d]) (logid:0ac1b254) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 04:16:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-f539f69c]) (logid:8c1ab635) DbCollector is running... 2025-06-29 04:16:43,094 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-7345a91a]) (logid:9569792d) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:16:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-7345a91a]) (logid:9569792d) Snapshot scheduler is being called at [2025-06-29T02:16:43+0000]. 2025-06-29 04:16:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-7345a91a]) (logid:9569792d) There are [0] scheduled snapshots to be executed at [2025-06-29T02:16:43+0000]. 2025-06-29 04:16:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-2a586a1e]) (logid:069cb1e4) Found 0 running routers. 2025-06-29 04:16:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2c1ce0a0]) (logid:0ad56c5c) Found 0 routers to update status. 2025-06-29 04:16:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2c1ce0a0]) (logid:0ad56c5c) Found 0 VPC's to update Redundant State. 2025-06-29 04:16:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2c1ce0a0]) (logid:0ad56c5c) Found 0 networks to update RvR status. 2025-06-29 04:16:43,110 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Found 1 running routers. 2025-06-29 04:16:43,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:16:43,116 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Seq 2-3926857400090362043: Routed from 90520738888109 2025-06-29 04:16:43,116 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Seq 2-3926857400090362043: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:43,118 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-ee9549f3]) (logid:e848b327) External devices are not used. Skipping external device usage collection 2025-06-29 04:16:43,130 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-3fbcfd2d]) (logid:c30adf04) Started resource counters recalculation periodic task. 2025-06-29 04:16:43,132 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-3fbcfd2d]) (logid:c30adf04) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:16:43,306 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-a36d0f40]) (logid:6516ac27) Backup backup.poll is being called at 2025-06-29T02:16:43+0000 2025-06-29 04:16:43,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f25ee3c2]) (logid:a3978618) No inactive management server node found 2025-06-29 04:16:43,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f25ee3c2]) (logid:a3978618) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:43,523 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-3fbcfd2d]) (logid:c30adf04) Finished resource counters recalculation periodic task. 2025-06-29 04:16:43,646 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 2-3926857400090362043: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:43,646 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Seq 2-3926857400090362043: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:16:43,646 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 04:16:43,650 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 04:16:43,694 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 04:16:43,696 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Seq 2-3926857400090362044: Routed from 90520738888109 2025-06-29 04:16:43,697 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Seq 2-3926857400090362044: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) ManagementServerCollector is running... 2025-06-29 04:16:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) DB properties were already loaded 2025-06-29 04:16:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:16:44,055 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Execution of process [4066506] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:16:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Exit value of process [4066506] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:16:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Process [4066506] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:16:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) usage install: null 2025-06-29 04:16:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Sessions active: 0 2025-06-29 04:16:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:16:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066509] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:16:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:16:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066514] for command [/bin/bash -c uname -r ]. 2025-06-29 04:16:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:16:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066515] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:16:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) system memory from /proc: 4887515136 2025-06-29 04:16:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:16:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066520] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:16:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) free memory from /proc: 297136128 2025-06-29 04:16:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:16:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066525] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:16:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) used memory from /proc: 1070588 2025-06-29 04:16:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:16:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066526] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:16:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:16:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066527] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:16:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:16:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066528] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:16:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:16:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066535] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:16:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:16:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Successfully executed process [4066536] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:16:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:16:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail buffersdirect.count, 93 2025-06-29 04:16:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail buffersdirect.used, 321663 2025-06-29 04:16:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail buffersmapped.capacity, 0 2025-06-29 04:16:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail buffersmapped.count, 0 2025-06-29 04:16:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail buffersmapped.used, 0 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail jvmuptime, 2463833 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memoryheap.committed, 620232704 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memoryheap.init, 77594624 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memoryheap.usage, 0.13855234432665733 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorynon-heap.committed, 196149248 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorynon-heap.max, -1 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorynon-heap.usage, -1.93799752E8 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorynon-heap.used, 193799752 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11270327912556652 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21721285866631118 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.Metaspace.usage, 0.9931083459984397 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.PS-Eden-Space.usage, 0.12718828986672795 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorytotal.committed, 816381952 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorytotal.init, 85262336 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorytotal.max, 1908932607 2025-06-29 04:16:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail memorytotal.used, 458286840 2025-06-29 04:16:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail threadsdeadlocks, [] 2025-06-29 04:16:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail threadsnew.count, 0 2025-06-29 04:16:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:16:44,134 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-7f2dd099]) (logid:1521a04c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:16:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":356476960,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2463759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":457551016,"totalCommitted":816381952,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":264487088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297136128,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.4,0.54],"systemCyclesUsage":[66411201,53832386,4589553629],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:16:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 81, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:16:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":356476960,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2463759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":457551016,"totalCommitted":816381952,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":264487088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297136128,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.4,0.54],"systemCyclesUsage":[66411201,53832386,4589553629],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:16:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:16:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-19:[ctx-6496f390]) (logid:f19d866f) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:16:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":356476960,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2463759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":457551016,"totalCommitted":816381952,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":264487088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297136128,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.4,0.54],"systemCyclesUsage":[66411201,53832386,4589553629],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:16:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:16:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 81, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:16:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":356476960,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2463759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.17,"totalInit":85262336,"totalUsed":457551016,"totalCommitted":816381952,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":264487088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297136128,"systemMemoryUsed":1070588,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.17,0.4,0.54],"systemCyclesUsage":[66411201,53832386,4589553629],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:16:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65c2517b]) (logid:7e82afcc) No inactive management server node found 2025-06-29 04:16:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65c2517b]) (logid:7e82afcc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:45,127 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 2-3926857400090362044: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.d99a9355-f29a-49f3-9575-0452026b004c","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:45,127 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-8790675d]) (logid:15587549) Seq 2-3926857400090362044: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 04:16:45,130 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 04:16:45,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:16:45,134 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Seq 2-3926857400090362045: Routed from 90520738888109 2025-06-29 04:16:45,135 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Seq 2-3926857400090362045: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:45,313 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8539: Seq 7-8539: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:45,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8539 2025-06-29 04:16:45,588 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 2-3926857400090362045: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "8.41665267944336", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "17.192840576171875", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "26.875972747802734", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "36.73601150512695", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163301560", "lastRunDuration": "41.48674011230469", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163301626", "lastRunDuration": "8070.9991455078125", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163309736", "lastRunDuration": "54.15153503417969", "message": "Memory Usage within limits with current at 50.8766%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163309791", "lastRunDuration": "3220.878839492798", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163313056", "lastRunDuration": "41.91732406616211", "message": "Sufficient free space is 2571.33203125 MB"}, "lastRun": {"start": "2025-06-29 02:15:01.415527", "end": "2025-06-29 02:15:13.098514", "duration": "11.682987213134766"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163001964", "lastRunDuration": "58.664798736572266", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163002023", "lastRunDuration": "71.38872146606445", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163002228", "lastRunDuration": "41.3820743560791", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163002402", "lastRunDuration": "43.996572494506836", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:10:01.960852", "end": "2025-06-29 02:10:02.496083", "duration": "0.5352308750152588"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "8.41665267944336", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "17.192840576171875", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "26.875972747802734", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "36.73601150512695", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163301560", "lastRunDuration": "41.48674011230469", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163301626", "lastRunDuration": "8070.9991455078125", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163309736", "lastRunDuration": "54.15153503417969", "message": "Memory Usage within limits with current at 50.8766%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163309791", "lastRunDuration": "3220.878839492798", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163313056", "lastRunDuration": "41.91732406616211", "message": "Sufficient free space is 2571.33203125 MB"}, "lastRun": {"start": "2025-06-29 02:15:01.415527", "end": "2025-06-29 02:15:13.098514", "duration": "11.682987213134766"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163001964", "lastRunDuration": "58.664798736572266", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163002023", "lastRunDuration": "71.38872146606445", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163002228", "lastRunDuration": "41.3820743560791", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163002402", "lastRunDuration": "43.996572494506836", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:10:01.960852", "end": "2025-06-29 02:10:02.496083", "duration": "0.5352308750152588"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:45,588 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Seq 2-3926857400090362045: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:16:45,588 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "8.41665267944336", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "17.192840576171875", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "26.875972747802734", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "36.73601150512695", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163301560", "lastRunDuration": "41.48674011230469", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163301626", "lastRunDuration": "8070.9991455078125", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163309736", "lastRunDuration": "54.15153503417969", "message": "Memory Usage within limits with current at 50.8766%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163309791", "lastRunDuration": "3220.878839492798", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163313056", "lastRunDuration": "41.91732406616211", "message": "Sufficient free space is 2571.33203125 MB"}, "lastRun": {"start": "2025-06-29 02:15:01.415527", "end": "2025-06-29 02:15:13.098514", "duration": "11.682987213134766"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163001964", "lastRunDuration": "58.664798736572266", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163002023", "lastRunDuration": "71.38872146606445", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163002228", "lastRunDuration": "41.3820743560791", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163002402", "lastRunDuration": "43.996572494506836", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:10:01.960852", "end": "2025-06-29 02:10:02.496083", "duration": "0.5352308750152588"}}} 2025-06-29 04:16:45,618 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "8.41665267944336", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "17.192840576171875", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "26.875972747802734", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163301417", "lastRunDuration": "36.73601150512695", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163301560", "lastRunDuration": "41.48674011230469", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163301626", "lastRunDuration": "8070.9991455078125", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163309736", "lastRunDuration": "54.15153503417969", "message": "Memory Usage within limits with current at 50.8766%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163309791", "lastRunDuration": "3220.878839492798", "message": "CPU Usage within limits with current at 0.3%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163313056", "lastRunDuration": "41.91732406616211", "message": "Sufficient free space is 2571.33203125 MB"}, "lastRun": {"start": "2025-06-29 02:15:01.415527", "end": "2025-06-29 02:15:13.098514", "duration": "11.682987213134766"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163001964", "lastRunDuration": "58.664798736572266", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163002023", "lastRunDuration": "71.38872146606445", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163002228", "lastRunDuration": "41.3820743560791", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163002402", "lastRunDuration": "43.996572494506836", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:10:01.960852", "end": "2025-06-29 02:10:02.496083", "duration": "0.5352308750152588"}}} 2025-06-29 04:16:45,620 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found check types executed on VR basic, start: 2025-06-29 02:15:01.415527, end: 2025-06-29 02:15:13.098514, duration: 11.682987213134766 2025-06-29 04:16:45,629 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6a1e0d36- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 04:15:01 CEST 2025, details: service is running which took running duration (ms) 8.41665267944336 2025-06-29 04:16:45,640 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@c936784- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 04:15:01 CEST 2025, details: service is running which took running duration (ms) 17.192840576171875 2025-06-29 04:16:45,650 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7e60d64c- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 04:15:01 CEST 2025, details: service is running which took running duration (ms) 26.875972747802734 2025-06-29 04:16:45,662 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4409510b- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 04:15:01 CEST 2025, details: service is running which took running duration (ms) 36.73601150512695 2025-06-29 04:16:45,672 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5e648301- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 04:15:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 41.48674011230469 2025-06-29 04:16:45,683 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@495c700e- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 04:15:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8070.9991455078125 2025-06-29 04:16:45,694 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@31f61ddf- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 04:15:09 CEST 2025, details: Memory Usage within limits with current at 50.8766% which took running duration (ms) 54.15153503417969 2025-06-29 04:16:45,705 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@3ff993e3- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 04:15:09 CEST 2025, details: CPU Usage within limits with current at 0.3% which took running duration (ms) 3220.878839492798 2025-06-29 04:16:45,715 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@457f7396- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 04:15:13 CEST 2025, details: Sufficient free space is 2571.33203125 MB which took running duration (ms) 41.91732406616211 2025-06-29 04:16:45,715 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found check types executed on VR advanced, start: 2025-06-29 02:10:01.960852, end: 2025-06-29 02:10:02.496083, duration: 0.5352308750152588 2025-06-29 04:16:45,725 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6f09a737- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 04:10:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 58.664798736572266 2025-06-29 04:16:45,736 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6e9047ec- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 04:10:02 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 71.38872146606445 2025-06-29 04:16:45,747 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1d85a756- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 04:10:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 41.3820743560791 2025-06-29 04:16:45,758 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a271d9c3]) (logid:7389f0d4) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@23d884b3- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 04:10:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 43.996572494506836 2025-06-29 04:16:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:16:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:16:46,212 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f833486d]) (logid:c5447208) HA health check task is running... 2025-06-29 04:16:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e1d63f8]) (logid:e99182dc) No inactive management server node found 2025-06-29 04:16:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e1d63f8]) (logid:e99182dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:16:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:16:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:16:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:16:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:16:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:16:47,074 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:16:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:16:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d2f41df]) (logid:9fca6d08) No inactive management server node found 2025-06-29 04:16:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d2f41df]) (logid:9fca6d08) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:48,049 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-5:[ctx-4542b71c]) (logid:f4c8a06a) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:16:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-89b192fb]) (logid:cf373c96) VM scheduler.poll is being called at 2025-06-29T02:17:00+0000 2025-06-29 04:16:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-89b192fb]) (logid:cf373c96) Got 0 scheduled jobs to be executed at 2025-06-29T02:17:00+0000 2025-06-29 04:16:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-89b192fb]) (logid:cf373c96) Cleaned up 0 VM scheduled job entries 2025-06-29 04:16:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d927ca35]) (logid:0abdb090) VM Operation Thread Running 2025-06-29 04:16:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e01657eb]) (logid:906e7c53) ClusterDRS.poll is being called at 2025-06-29T02:17:00+0000 2025-06-29 04:16:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e01657eb]) (logid:906e7c53) Removed 0 old drs migration plans 2025-06-29 04:16:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d5dbc96]) (logid:1cab4474) No inactive management server node found 2025-06-29 04:16:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d5dbc96]) (logid:1cab4474) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:50,213 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e4eadc76]) (logid:6282fa18) HA health check task is running... 2025-06-29 04:16:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06b1e131]) (logid:5e8e49e6) No inactive management server node found 2025-06-29 04:16:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06b1e131]) (logid:5e8e49e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c21f3408]) (logid:0f10ecc7) No inactive management server node found 2025-06-29 04:16:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c21f3408]) (logid:0f10ecc7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:53,151 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-e56862c0]) (logid:b0cc6b91) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:16:53,161 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-e56862c0]) (logid:b0cc6b91) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:16:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-386d8117]) (logid:14e2d991) No inactive management server node found 2025-06-29 04:16:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-386d8117]) (logid:14e2d991) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:54,215 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7d4d20dd]) (logid:4bcec69c) HA health check task is running... 2025-06-29 04:16:55,313 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8541: Seq 7-8541: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:16:55,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8541 2025-06-29 04:16:55,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9655d589]) (logid:9a69062f) No inactive management server node found 2025-06-29 04:16:55,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9655d589]) (logid:9a69062f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-809b3644]) (logid:5a4204cc) No inactive management server node found 2025-06-29 04:16:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-809b3644]) (logid:5a4204cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:58,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bc245ed6]) (logid:d7a84fc3) Enabled non-edge zones available for scan: 1 2025-06-29 04:16:58,097 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:16:58,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bc245ed6]) (logid:d7a84fc3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:16:58,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bc245ed6]) (logid:d7a84fc3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:16:58,101 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Seq 2-4725683383995203758: Routed from 90520738888109 2025-06-29 04:16:58,145 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Seq 1-4725683383995203758: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:16:58,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 04:16:58,177 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:16:58,179 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Seq 2-3926857400090362046: Routed from 90520738888109 2025-06-29 04:16:58,216 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1ad3b5dc]) (logid:96f2e3c6) HA health check task is running... 2025-06-29 04:16:58,223 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Seq 2-3926857400090362046: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:16:58,224 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0299e71f]) (logid:f299e502) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 04:16:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6d224583]) (logid:14c8d8f2) Enabled non-edge zones available for scan: 1 2025-06-29 04:16:58,283 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-6d224583]) (logid:14c8d8f2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:16:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebd117bb]) (logid:c0c552aa) No inactive management server node found 2025-06-29 04:16:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebd117bb]) (logid:c0c552aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:16:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8efce07]) (logid:c9c4d63b) No inactive management server node found 2025-06-29 04:16:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8efce07]) (logid:c9c4d63b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63fc1591]) (logid:213b9a1c) No inactive management server node found 2025-06-29 04:17:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63fc1591]) (logid:213b9a1c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:02,217 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-916362a9]) (logid:17b8642c) HA health check task is running... 2025-06-29 04:17:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b32be0f]) (logid:2637c418) No inactive management server node found 2025-06-29 04:17:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b32be0f]) (logid:2637c418) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f768f30]) (logid:a0b3abb6) No inactive management server node found 2025-06-29 04:17:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f768f30]) (logid:a0b3abb6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:05,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8542: Seq 7-8542: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:17:05,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8542 2025-06-29 04:17:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e248e16b]) (logid:ebc14931) No inactive management server node found 2025-06-29 04:17:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e248e16b]) (logid:ebc14931) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:06,218 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-10ffba46]) (logid:f2429544) HA health check task is running... 2025-06-29 04:17:07,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a1b1a95]) (logid:c504f2f2) No inactive management server node found 2025-06-29 04:17:07,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a1b1a95]) (logid:c504f2f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef1cd765]) (logid:6a2c5d14) No inactive management server node found 2025-06-29 04:17:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef1cd765]) (logid:6a2c5d14) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:10,220 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3a8553ac]) (logid:3049f000) HA health check task is running... 2025-06-29 04:17:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d948603]) (logid:6e0b46bf) No inactive management server node found 2025-06-29 04:17:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d948603]) (logid:6e0b46bf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f944e775]) (logid:85cb26c1) No inactive management server node found 2025-06-29 04:17:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f944e775]) (logid:85cb26c1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:13,003 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-20fcdf53]) (logid:d07ca249) Checking if any host reservation can be released ... 2025-06-29 04:17:13,009 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-20fcdf53]) (logid:d07ca249) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:13,012 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-20fcdf53]) (logid:d07ca249) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:13,012 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-20fcdf53]) (logid:d07ca249) Done running HostReservationReleaseChecker ... 2025-06-29 04:17:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-48667837]) (logid:d7b0fd57) Found 0 routers to update status. 2025-06-29 04:17:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-48667837]) (logid:d7b0fd57) Found 0 VPC's to update Redundant State. 2025-06-29 04:17:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-48667837]) (logid:d7b0fd57) Found 0 networks to update RvR status. 2025-06-29 04:17:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Running Capacity Checker ... 2025-06-29 04:17:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) recalculating system capacity 2025-06-29 04:17:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Executing cpu/ram capacity update 2025-06-29 04:17:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:13,183 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:17:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:17:13,189 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:17:13,201 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:13,202 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:13,210 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:17:13,213 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:17:13,213 DEBUG [c.c.c.CapacityManagerImpl] (pool-28-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:17:13,222 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Done executing cpu/ram capacity update 2025-06-29 04:17:13,222 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Executing storage capacity update 2025-06-29 04:17:13,231 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:17:13,232 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:17:13,232 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:17:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:17:13,242 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:17:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:17:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:17:13,245 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:17:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:17:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:17:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:17:13,260 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:17:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:17:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:17:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:17:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-29-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:17:13,272 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Done executing storage capacity update 2025-06-29 04:17:13,272 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Executing capacity updates for public ip and Vlans 2025-06-29 04:17:13,286 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Done capacity updates for public ip and Vlans 2025-06-29 04:17:13,286 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Executing capacity updates for private ip 2025-06-29 04:17:13,291 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Done executing capacity updates for private ip 2025-06-29 04:17:13,291 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Done recalculating system capacity 2025-06-29 04:17:13,332 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-39705d4a]) (logid:b5ec692f) Done running Capacity Checker ... 2025-06-29 04:17:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20011ff5]) (logid:3d2261d9) No inactive management server node found 2025-06-29 04:17:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20011ff5]) (logid:3d2261d9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:14,221 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-43fe3146]) (logid:007497d2) HA health check task is running... 2025-06-29 04:17:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6d56536]) (logid:fe105c25) No inactive management server node found 2025-06-29 04:17:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6d56536]) (logid:fe105c25) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:15,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8543: Seq 7-8543: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:17:15,325 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8543 2025-06-29 04:17:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6811eb0]) (logid:17f045a2) No inactive management server node found 2025-06-29 04:17:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6811eb0]) (logid:17f045a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-402c33ef]) (logid:4720333a) No inactive management server node found 2025-06-29 04:17:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-402c33ef]) (logid:4720333a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:18,011 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-47eed34d]) (logid:3a366d67) AutoScaling Monitor is running... 2025-06-29 04:17:18,073 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-69bd5145]) (logid:0904cbbd) Starting to remove all volume_stats rows older than [Sat Jun 28 16:17:18 CEST 2025]. 2025-06-29 04:17:18,074 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-69bd5145]) (logid:0904cbbd) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:17:18 CEST 2025]. 2025-06-29 04:17:18,091 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8ae0ea94]) (logid:7d68a380) Resetting hosts suitable for reconnect 2025-06-29 04:17:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8ae0ea94]) (logid:7d68a380) Completed resetting hosts suitable for reconnect 2025-06-29 04:17:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8ae0ea94]) (logid:7d68a380) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:17:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8ae0ea94]) (logid:7d68a380) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:17:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8ae0ea94]) (logid:7d68a380) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:17:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8ae0ea94]) (logid:7d68a380) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:17:18,222 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-11cf5f4e]) (logid:a68c036d) HA health check task is running... 2025-06-29 04:17:18,373 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-6f03e64e]) (logid:1c275935) Starting to remove all vm_stats rows older than [Sat Jun 28 16:17:18 CEST 2025]. 2025-06-29 04:17:18,397 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-6f03e64e]) (logid:1c275935) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:17:18 CEST 2025]. 2025-06-29 04:17:19,076 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:17:19,078 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:17:19,079 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) Seq 2-4725683383995203759: Routed from 90520738888109 2025-06-29 04:17:19,122 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) Seq 1-4725683383995203759: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:17:19,123 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:17:19,124 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) Seq 2-3926857400090362047: Routed from 90520738888109 2025-06-29 04:17:19,168 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-c670f8ce]) (logid:3d7e2d6a) Seq 2-3926857400090362047: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:17:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cf44e8a]) (logid:d6dad2d3) No inactive management server node found 2025-06-29 04:17:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cf44e8a]) (logid:d6dad2d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fc0da68]) (logid:30f43445) No inactive management server node found 2025-06-29 04:17:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fc0da68]) (logid:30f43445) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:20,930 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:17:20,933 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:17:20,934 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) Seq 2-4725683383995203760: Routed from 90520738888109 2025-06-29 04:17:20,986 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) Seq 1-4725683383995203760: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:17:21,025 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:17:21,026 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) Seq 2-3926857400090362048: Routed from 90520738888109 2025-06-29 04:17:21,115 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-af711ae7]) (logid:54602d51) Seq 2-3926857400090362048: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:17:21,422 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) StorageCollector is running... 2025-06-29 04:17:21,429 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:17:21,429 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:17:21,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:17:21,430 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-3938116399158788144: Routed from 90520738888109 2025-06-29 04:17:21,505 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 4-3938116399158788144: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:17:21,510 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:17:21,510 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:17:21,510 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:17:21,512 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-3926857400090362049: Routed from 90520738888109 2025-06-29 04:17:21,526 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-3926857400090362049: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:17:21,537 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:17:21,537 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:17:21,537 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:17:21,538 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-4725683383995203761: Routed from 90520738888109 2025-06-29 04:17:21,586 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 1-4725683383995203761: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:17:21,598 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:17:21,598 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:17:21,598 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:17:21,599 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-4725683383995203762: Routed from 90520738888109 2025-06-29 04:17:21,684 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 1-4725683383995203762: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:17:21,686 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:17:21,686 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:17:21,686 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:17:21,687 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-3926857400090362050: Routed from 90520738888109 2025-06-29 04:17:21,731 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-77cc3978]) (logid:3d56f0d6) Seq 2-3926857400090362050: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:17:22,223 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ded3a686]) (logid:5b2e97e7) HA health check task is running... 2025-06-29 04:17:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5f28807]) (logid:12c04525) No inactive management server node found 2025-06-29 04:17:22,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5f28807]) (logid:12c04525) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-656a1156]) (logid:c28c86af) No inactive management server node found 2025-06-29 04:17:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-656a1156]) (logid:c28c86af) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:17:25,314 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8544: Seq 7-8544: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:17:25,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8544 2025-06-29 04:17:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18963d31]) (logid:03b66242) No inactive management server node found 2025-06-29 04:17:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18963d31]) (logid:03b66242) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:26,225 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7b2dc722]) (logid:058b15fc) HA health check task is running... 2025-06-29 04:17:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a31d137]) (logid:002eb51f) No inactive management server node found 2025-06-29 04:17:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a31d137]) (logid:002eb51f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:28,091 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fb1bf4af]) (logid:5ee93393) Enabled non-edge zones available for scan: 1 2025-06-29 04:17:28,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-fb1bf4af]) (logid:5ee93393) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:17:28,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fb1bf4af]) (logid:5ee93393) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:17:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d74d96fc]) (logid:79d40ecb) Enabled non-edge zones available for scan: 1 2025-06-29 04:17:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d74d96fc]) (logid:79d40ecb) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:17:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e88ea2fd]) (logid:f7de4b25) No inactive management server node found 2025-06-29 04:17:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e88ea2fd]) (logid:f7de4b25) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6009579f]) (logid:00c3962f) No inactive management server node found 2025-06-29 04:17:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6009579f]) (logid:00c3962f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:30,226 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d14de3d1]) (logid:3f6dd678) HA health check task is running... 2025-06-29 04:17:31,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-365ed588]) (logid:ea4076e1) No inactive management server node found 2025-06-29 04:17:31,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-365ed588]) (logid:ea4076e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-70c6a7e9]) (logid:bc794cc9) No inactive management server node found 2025-06-29 04:17:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-70c6a7e9]) (logid:bc794cc9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:34,228 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-08c14026]) (logid:e3e5ff40) HA health check task is running... 2025-06-29 04:17:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-509676be]) (logid:9d423a01) No inactive management server node found 2025-06-29 04:17:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-509676be]) (logid:9d423a01) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:35,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8545: Seq 7-8545: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:17:35,325 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8545 2025-06-29 04:17:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bedb75dc]) (logid:706120e5) No inactive management server node found 2025-06-29 04:17:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bedb75dc]) (logid:706120e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2eecf692]) (logid:68c1b205) No inactive management server node found 2025-06-29 04:17:37,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2eecf692]) (logid:68c1b205) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:38,229 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-69659102]) (logid:cbfa8c1d) HA health check task is running... 2025-06-29 04:17:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40275930]) (logid:438d591b) No inactive management server node found 2025-06-29 04:17:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40275930]) (logid:438d591b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ced3573]) (logid:3efd0827) No inactive management server node found 2025-06-29 04:17:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ced3573]) (logid:3efd0827) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:41,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bd5f4f8]) (logid:7db8cbbb) No inactive management server node found 2025-06-29 04:17:41,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bd5f4f8]) (logid:7db8cbbb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:42,231 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-944ee366]) (logid:b8fcf302) HA health check task is running... 2025-06-29 04:17:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-32cba2ff]) (logid:0378bb12) DbCollector is running... 2025-06-29 04:17:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a0a0d3f8]) (logid:fb2474f7) Found 0 routers to update status. 2025-06-29 04:17:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a0a0d3f8]) (logid:fb2474f7) Found 0 VPC's to update Redundant State. 2025-06-29 04:17:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a0a0d3f8]) (logid:fb2474f7) Found 0 networks to update RvR status. 2025-06-29 04:17:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc585e7]) (logid:fc6547f3) No inactive management server node found 2025-06-29 04:17:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bc585e7]) (logid:fc6547f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) ManagementServerCollector is running... 2025-06-29 04:17:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) DB properties were already loaded 2025-06-29 04:17:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:17:44,062 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Execution of process [4066543] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:17:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Exit value of process [4066543] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:17:44,062 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Process [4066543] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:17:44,062 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) usage install: null 2025-06-29 04:17:44,062 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Sessions active: 0 2025-06-29 04:17:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:17:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066546] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:17:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:17:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066551] for command [/bin/bash -c uname -r ]. 2025-06-29 04:17:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:17:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066552] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:17:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) system memory from /proc: 4887515136 2025-06-29 04:17:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:17:44,087 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066557] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:17:44,087 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) free memory from /proc: 302116864 2025-06-29 04:17:44,087 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:17:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066562] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:17:44,100 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) used memory from /proc: 1070844 2025-06-29 04:17:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:17:44,105 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066563] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:17:44,105 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:17:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066564] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:17:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:17:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066565] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:17:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:17:44,138 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066572] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:17:44,139 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:17:44,163 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Successfully executed process [4066573] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:17:44,179 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail buffersdirect.count, 93 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail buffersdirect.used, 321663 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail buffersmapped.capacity, 0 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail buffersmapped.count, 0 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail buffersmapped.used, 0 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail jvmuptime, 2523883 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memoryheap.committed, 620232704 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memoryheap.init, 77594624 2025-06-29 04:17:44,180 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memoryheap.usage, 0.15868144047126048 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorynon-heap.committed, 196345856 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorynon-heap.max, -1 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorynon-heap.usage, -1.94019024E8 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorynon-heap.used, 194019024 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11363740502532658 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2179428966574466 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.Metaspace.usage, 0.9932142016015073 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.PS-Eden-Space.usage, 0.18544746241243543 2025-06-29 04:17:44,181 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:17:44,194 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:17:44,194 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorytotal.committed, 816578560 2025-06-29 04:17:44,194 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorytotal.init, 85262336 2025-06-29 04:17:44,194 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorytotal.max, 1908932607 2025-06-29 04:17:44,194 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail memorytotal.used, 496931200 2025-06-29 04:17:44,195 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail threadsdeadlocks, [] 2025-06-29 04:17:44,196 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail threadsnew.count, 0 2025-06-29 04:17:44,197 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) not storing detail threadstimed_waiting.count, 82 2025-06-29 04:17:44,210 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-a9d6bc7d]) (logid:57114643) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:17:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":318005040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2523765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":496246688,"totalCommitted":816578560,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302912176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302116864,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.33,0.5],"systemCyclesUsage":[66411596,53832700,4589576678],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:17:44,211 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 83, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:17:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":318005040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2523765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":496246688,"totalCommitted":816578560,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302912176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302116864,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.33,0.5],"systemCyclesUsage":[66411596,53832700,4589576678],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:17:44,211 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:17:44,227 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 15 ms 2025-06-29 04:17:44,227 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 16ms. agent: 0, pdu seq: 83, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:17:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":318005040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2523765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":496246688,"totalCommitted":816578560,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302912176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302116864,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.33,0.5],"systemCyclesUsage":[66411596,53832700,4589576678],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:17:44,228 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-20:[ctx-6acfe9c6]) (logid:130ed184) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:17:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":318005040,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2523765,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":496246688,"totalCommitted":816578560,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":302912176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":200,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":302116864,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.33,0.5],"systemCyclesUsage":[66411596,53832700,4589576678],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:17:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3996f291]) (logid:e41c7202) No inactive management server node found 2025-06-29 04:17:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3996f291]) (logid:e41c7202) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:45,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8546: Seq 7-8546: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:17:45,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8546 2025-06-29 04:17:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:17:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:17:46,232 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-5ecd44a4]) (logid:c5e20520) HA health check task is running... 2025-06-29 04:17:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f3c7491]) (logid:d543be1e) No inactive management server node found 2025-06-29 04:17:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f3c7491]) (logid:d543be1e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:17:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:17:47,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:17:47,087 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:17:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd4e78e7]) (logid:ed3c8dcd) No inactive management server node found 2025-06-29 04:17:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd4e78e7]) (logid:ed3c8dcd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-aed294d9]) (logid:8bd3111e) VM scheduler.poll is being called at 2025-06-29T02:18:00+0000 2025-06-29 04:17:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-aed294d9]) (logid:8bd3111e) Got 0 scheduled jobs to be executed at 2025-06-29T02:18:00+0000 2025-06-29 04:17:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-aed294d9]) (logid:8bd3111e) Cleaned up 0 VM scheduled job entries 2025-06-29 04:17:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ea8885eb]) (logid:abbcd3d4) VM Operation Thread Running 2025-06-29 04:17:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-96e4162e]) (logid:f74f8d09) ClusterDRS.poll is being called at 2025-06-29T02:18:00+0000 2025-06-29 04:17:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-96e4162e]) (logid:f74f8d09) Removed 0 old drs migration plans 2025-06-29 04:17:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239269ae]) (logid:a8874413) No inactive management server node found 2025-06-29 04:17:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239269ae]) (logid:a8874413) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:50,233 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-afee87fb]) (logid:a6363ed1) HA health check task is running... 2025-06-29 04:17:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-482aab20]) (logid:bb716ab7) No inactive management server node found 2025-06-29 04:17:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-482aab20]) (logid:bb716ab7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a5c6f1b]) (logid:8b40bbc4) No inactive management server node found 2025-06-29 04:17:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a5c6f1b]) (logid:8b40bbc4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d96c052e]) (logid:345a229a) No inactive management server node found 2025-06-29 04:17:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d96c052e]) (logid:345a229a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:54,235 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9b8f1bf8]) (logid:c57cbc04) HA health check task is running... 2025-06-29 04:17:55,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8548: Seq 7-8548: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:17:55,325 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8548 2025-06-29 04:17:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83897857]) (logid:22bb30a1) No inactive management server node found 2025-06-29 04:17:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83897857]) (logid:22bb30a1) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62db2426]) (logid:e4ddafb2) No inactive management server node found 2025-06-29 04:17:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62db2426]) (logid:e4ddafb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1646f487]) (logid:3deab381) Enabled non-edge zones available for scan: 1 2025-06-29 04:17:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1646f487]) (logid:3deab381) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:17:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1646f487]) (logid:3deab381) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:17:58,236 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9030fb38]) (logid:e9f908d1) HA health check task is running... 2025-06-29 04:17:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3ab6082e]) (logid:32f745ce) Enabled non-edge zones available for scan: 1 2025-06-29 04:17:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3ab6082e]) (logid:32f745ce) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:17:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39f0a1b1]) (logid:66130a4f) No inactive management server node found 2025-06-29 04:17:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39f0a1b1]) (logid:66130a4f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:17:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c5d061c]) (logid:493c04c7) No inactive management server node found 2025-06-29 04:17:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c5d061c]) (logid:493c04c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94249632]) (logid:8b51aeb2) No inactive management server node found 2025-06-29 04:18:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94249632]) (logid:8b51aeb2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:02,238 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c115e767]) (logid:e8e00b42) HA health check task is running... 2025-06-29 04:18:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a55ecb5e]) (logid:aa60a239) No inactive management server node found 2025-06-29 04:18:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a55ecb5e]) (logid:aa60a239) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2092e1b]) (logid:72ffc51b) No inactive management server node found 2025-06-29 04:18:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2092e1b]) (logid:72ffc51b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:05,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8549: Seq 7-8549: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:18:05,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8549 2025-06-29 04:18:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3958f174]) (logid:7b81ac50) No inactive management server node found 2025-06-29 04:18:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3958f174]) (logid:7b81ac50) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:06,239 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-40bdb03d]) (logid:89077274) HA health check task is running... 2025-06-29 04:18:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8eb7d465]) (logid:cc4c09a8) No inactive management server node found 2025-06-29 04:18:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8eb7d465]) (logid:cc4c09a8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e56740f]) (logid:13615d43) No inactive management server node found 2025-06-29 04:18:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e56740f]) (logid:13615d43) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:10,241 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e1858053]) (logid:86c70ed1) HA health check task is running... 2025-06-29 04:18:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6202d57]) (logid:06aa334a) No inactive management server node found 2025-06-29 04:18:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6202d57]) (logid:06aa334a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cccfc35]) (logid:c82c5135) No inactive management server node found 2025-06-29 04:18:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cccfc35]) (logid:c82c5135) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6ca5135e]) (logid:5c0ad20f) Found 0 routers to update status. 2025-06-29 04:18:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6ca5135e]) (logid:5c0ad20f) Found 0 VPC's to update Redundant State. 2025-06-29 04:18:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6ca5135e]) (logid:5c0ad20f) Found 0 networks to update RvR status. 2025-06-29 04:18:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06f7f4ca]) (logid:e11c69d7) No inactive management server node found 2025-06-29 04:18:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06f7f4ca]) (logid:e11c69d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:14,242 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9a2fb9ea]) (logid:f59d5cfd) HA health check task is running... 2025-06-29 04:18:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b77d31f3]) (logid:e83531eb) No inactive management server node found 2025-06-29 04:18:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b77d31f3]) (logid:e83531eb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:15,316 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8550: Seq 7-8550: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:18:15,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8550 2025-06-29 04:18:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c09a759]) (logid:be0eef1b) No inactive management server node found 2025-06-29 04:18:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c09a759]) (logid:be0eef1b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6100257a]) (logid:cb8e3ba1) No inactive management server node found 2025-06-29 04:18:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6100257a]) (logid:cb8e3ba1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:18,013 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-9975a07c]) (logid:16052bb0) AutoScaling Monitor is running... 2025-06-29 04:18:18,077 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-69149a7a]) (logid:061b5d16) Starting to remove all volume_stats rows older than [Sat Jun 28 16:18:18 CEST 2025]. 2025-06-29 04:18:18,077 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-69149a7a]) (logid:061b5d16) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:18:18 CEST 2025]. 2025-06-29 04:18:18,243 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-928685ca]) (logid:aacc640e) HA health check task is running... 2025-06-29 04:18:18,399 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-7553a0b7]) (logid:55866226) Starting to remove all vm_stats rows older than [Sat Jun 28 16:18:18 CEST 2025]. 2025-06-29 04:18:18,423 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-7553a0b7]) (logid:55866226) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:18:18 CEST 2025]. 2025-06-29 04:18:19,171 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:18:19,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:18:19,175 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) Seq 2-4725683383995203763: Routed from 90520738888109 2025-06-29 04:18:19,218 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) Seq 1-4725683383995203763: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:18:19,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:18:19,221 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) Seq 2-3926857400090362051: Routed from 90520738888109 2025-06-29 04:18:19,265 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-cd4e2da6]) (logid:9964d402) Seq 2-3926857400090362051: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:18:19,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-421d35b6]) (logid:6ef17b25) No inactive management server node found 2025-06-29 04:18:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-421d35b6]) (logid:6ef17b25) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-103ed6b9]) (logid:c996cd3f) No inactive management server node found 2025-06-29 04:18:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-103ed6b9]) (logid:c996cd3f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:21,169 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:18:21,172 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:18:21,173 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) Seq 2-4725683383995203764: Routed from 90520738888109 2025-06-29 04:18:21,227 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) Seq 1-4725683383995203764: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:18:21,265 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:18:21,266 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) Seq 2-3926857400090362052: Routed from 90520738888109 2025-06-29 04:18:21,383 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5591f91e]) (logid:04e0a0fc) Seq 2-3926857400090362052: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:18:21,732 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) StorageCollector is running... 2025-06-29 04:18:21,738 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:18:21,738 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:18:21,738 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:18:21,739 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 2-3938116399158788145: Routed from 90520738888109 2025-06-29 04:18:21,797 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 4-3938116399158788145: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:18:21,801 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:18:21,801 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:18:21,801 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:18:21,802 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 2-4725683383995203765: Routed from 90520738888109 2025-06-29 04:18:21,849 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 1-4725683383995203765: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:18:21,860 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:18:21,860 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:18:21,860 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:18:21,861 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 2-4725683383995203766: Routed from 90520738888109 2025-06-29 04:18:21,950 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 1-4725683383995203766: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:18:21,964 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:18:21,964 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:18:21,964 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:18:21,965 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 2-4725683383995203767: Routed from 90520738888109 2025-06-29 04:18:22,052 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 1-4725683383995203767: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:18:22,054 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:18:22,054 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:18:22,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:18:22,056 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 2-3926857400090362053: Routed from 90520738888109 2025-06-29 04:18:22,061 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-2f222b70]) (logid:9b265ff5) Seq 2-3926857400090362053: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:18:22,244 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c6f2719b]) (logid:7823d1af) HA health check task is running... 2025-06-29 04:18:22,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4062dae1]) (logid:0a0fc52d) No inactive management server node found 2025-06-29 04:18:22,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4062dae1]) (logid:0a0fc52d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18822489]) (logid:cb022fe3) No inactive management server node found 2025-06-29 04:18:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18822489]) (logid:cb022fe3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:25,316 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8551: Seq 7-8551: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:18:25,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8551 2025-06-29 04:18:25,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d2210fa]) (logid:be04856f) No inactive management server node found 2025-06-29 04:18:25,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d2210fa]) (logid:be04856f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:26,245 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-fe09e312]) (logid:7d983dc3) HA health check task is running... 2025-06-29 04:18:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c9b8c6f]) (logid:3f523040) No inactive management server node found 2025-06-29 04:18:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c9b8c6f]) (logid:3f523040) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-85995977]) (logid:be84e048) Enabled non-edge zones available for scan: 1 2025-06-29 04:18:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-85995977]) (logid:be84e048) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:18:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-85995977]) (logid:be84e048) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:18:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2bdf1c34]) (logid:b9450fbd) Enabled non-edge zones available for scan: 1 2025-06-29 04:18:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2bdf1c34]) (logid:b9450fbd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:18:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8522440b]) (logid:2f009ca2) No inactive management server node found 2025-06-29 04:18:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8522440b]) (logid:2f009ca2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd956739]) (logid:604bd5c7) No inactive management server node found 2025-06-29 04:18:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd956739]) (logid:604bd5c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:30,247 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-408a9fad]) (logid:b32eee73) HA health check task is running... 2025-06-29 04:18:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6637cd80]) (logid:5ef188fe) No inactive management server node found 2025-06-29 04:18:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6637cd80]) (logid:5ef188fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1205f4c]) (logid:2f76d2ab) No inactive management server node found 2025-06-29 04:18:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1205f4c]) (logid:2f76d2ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:34,249 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-364c604a]) (logid:fb36f802) HA health check task is running... 2025-06-29 04:18:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd658880]) (logid:286f7d98) No inactive management server node found 2025-06-29 04:18:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd658880]) (logid:286f7d98) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:35,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8552: Seq 7-8552: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:18:35,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8552 2025-06-29 04:18:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d143f795]) (logid:85db8a63) No inactive management server node found 2025-06-29 04:18:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d143f795]) (logid:85db8a63) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:37,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d474635]) (logid:428184e6) No inactive management server node found 2025-06-29 04:18:37,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d474635]) (logid:428184e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:38,250 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1f49077a]) (logid:15d60329) HA health check task is running... 2025-06-29 04:18:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de47dddf]) (logid:e01b0a4e) No inactive management server node found 2025-06-29 04:18:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de47dddf]) (logid:e01b0a4e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f6fdb8e]) (logid:f0023764) No inactive management server node found 2025-06-29 04:18:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f6fdb8e]) (logid:f0023764) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:41,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c59fe716]) (logid:a38694e6) No inactive management server node found 2025-06-29 04:18:41,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c59fe716]) (logid:a38694e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:42,251 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-091035ef]) (logid:55164733) HA health check task is running... 2025-06-29 04:18:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-95894c17]) (logid:f6a397b7) DbCollector is running... 2025-06-29 04:18:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fd06a979]) (logid:1e5340de) Found 0 routers to update status. 2025-06-29 04:18:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fd06a979]) (logid:1e5340de) Found 0 VPC's to update Redundant State. 2025-06-29 04:18:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fd06a979]) (logid:1e5340de) Found 0 networks to update RvR status. 2025-06-29 04:18:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd667571]) (logid:20e8d098) No inactive management server node found 2025-06-29 04:18:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd667571]) (logid:20e8d098) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) ManagementServerCollector is running... 2025-06-29 04:18:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) DB properties were already loaded 2025-06-29 04:18:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:18:44,055 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Execution of process [4066687] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:18:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Exit value of process [4066687] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:18:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Process [4066687] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:18:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) usage install: null 2025-06-29 04:18:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Sessions active: 0 2025-06-29 04:18:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:18:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066690] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:18:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:18:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066695] for command [/bin/bash -c uname -r ]. 2025-06-29 04:18:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:18:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066696] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:18:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) system memory from /proc: 4887515136 2025-06-29 04:18:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:18:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066701] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:18:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) free memory from /proc: 292667392 2025-06-29 04:18:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:18:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066706] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:18:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) used memory from /proc: 1070844 2025-06-29 04:18:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:18:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066707] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:18:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:18:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066708] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:18:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:18:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066709] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:18:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:18:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066716] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:18:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:18:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Successfully executed process [4066717] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:18:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:18:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail buffersdirect.count, 93 2025-06-29 04:18:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail buffersdirect.used, 321663 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail buffersmapped.capacity, 0 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail buffersmapped.count, 0 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail buffersmapped.used, 0 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail jvmuptime, 2583833 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memoryheap.committed, 620232704 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memoryheap.init, 77594624 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memoryheap.usage, 0.17028313238389609 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorynon-heap.committed, 196411392 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:18:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorynon-heap.max, -1 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorynon-heap.usage, -1.9408352E8 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorynon-heap.used, 194083520 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11377799253532392 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21824595094477955 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.Metaspace.usage, 0.993285845317652 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.PS-Eden-Space.usage, 0.2190259681786566 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorytotal.committed, 816644096 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorytotal.init, 85262336 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorytotal.max, 1908932607 2025-06-29 04:18:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail memorytotal.used, 519147648 2025-06-29 04:18:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail threadsdeadlocks, [] 2025-06-29 04:18:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail threadsnew.count, 0 2025-06-29 04:18:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:18:44,136 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-81e29a73]) (logid:29e442e4) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:18:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":295972192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2583759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":518338744,"totalCommitted":816644096,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":325059024,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292667392,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.3,0.48],"systemCyclesUsage":[66412560,53833096,4589599020],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:18:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 85, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:18:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":295972192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2583759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":518338744,"totalCommitted":816644096,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":325059024,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292667392,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.3,0.48],"systemCyclesUsage":[66412560,53833096,4589599020],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:18:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:18:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-20:[ctx-da27d5f8]) (logid:ee203931) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:18:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":295972192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2583759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":518338744,"totalCommitted":816644096,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":325059024,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292667392,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.3,0.48],"systemCyclesUsage":[66412560,53833096,4589599020],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:18:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:18:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 85, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:18:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":295972192,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2583759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":518338744,"totalCommitted":816644096,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":325059024,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292667392,"systemMemoryUsed":1070844,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.3,0.48],"systemCyclesUsage":[66412560,53833096,4589599020],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:18:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2ee3fa1]) (logid:6158686d) No inactive management server node found 2025-06-29 04:18:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2ee3fa1]) (logid:6158686d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:45,317 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8553: Seq 7-8553: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:18:45,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8553 2025-06-29 04:18:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:18:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:18:46,253 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a33aa5f3]) (logid:0b877bd0) HA health check task is running... 2025-06-29 04:18:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fec9f45b]) (logid:436dc432) No inactive management server node found 2025-06-29 04:18:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fec9f45b]) (logid:436dc432) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:18:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:18:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:18:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:18:47,070 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:18:47,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:18:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:18:47,075 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:18:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a193fc96]) (logid:0683fb1e) No inactive management server node found 2025-06-29 04:18:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a193fc96]) (logid:0683fb1e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ce6b78bb]) (logid:cf0ff3e3) Resetting hosts suitable for reconnect 2025-06-29 04:18:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ce6b78bb]) (logid:cf0ff3e3) Completed resetting hosts suitable for reconnect 2025-06-29 04:18:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ce6b78bb]) (logid:cf0ff3e3) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:18:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ce6b78bb]) (logid:cf0ff3e3) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:18:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ce6b78bb]) (logid:cf0ff3e3) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:18:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ce6b78bb]) (logid:cf0ff3e3) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:18:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f59bc9d0]) (logid:e35fbe1e) VM scheduler.poll is being called at 2025-06-29T02:19:00+0000 2025-06-29 04:18:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f59bc9d0]) (logid:e35fbe1e) Got 0 scheduled jobs to be executed at 2025-06-29T02:19:00+0000 2025-06-29 04:18:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f59bc9d0]) (logid:e35fbe1e) Cleaned up 0 VM scheduled job entries 2025-06-29 04:18:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-52dc9c29]) (logid:02d9d591) VM Operation Thread Running 2025-06-29 04:18:48,347 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-085f9071]) (logid:f7d25522) ClusterDRS.poll is being called at 2025-06-29T02:19:00+0000 2025-06-29 04:18:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-085f9071]) (logid:f7d25522) Removed 0 old drs migration plans 2025-06-29 04:18:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8039d8b6]) (logid:edaa7a5e) No inactive management server node found 2025-06-29 04:18:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8039d8b6]) (logid:edaa7a5e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:50,254 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e7da60d4]) (logid:90697bcd) HA health check task is running... 2025-06-29 04:18:50,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e821b83]) (logid:2ead2ce3) No inactive management server node found 2025-06-29 04:18:50,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e821b83]) (logid:2ead2ce3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d4dc15e]) (logid:eacf869d) No inactive management server node found 2025-06-29 04:18:52,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d4dc15e]) (logid:eacf869d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19236398]) (logid:a8736eb5) No inactive management server node found 2025-06-29 04:18:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19236398]) (logid:a8736eb5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:54,255 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e3f09731]) (logid:dcd44c78) HA health check task is running... 2025-06-29 04:18:55,318 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8555: Seq 7-8555: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:18:55,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8555 2025-06-29 04:18:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d103e700]) (logid:717027d2) No inactive management server node found 2025-06-29 04:18:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d103e700]) (logid:717027d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a35b9b6]) (logid:d4b118a0) No inactive management server node found 2025-06-29 04:18:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a35b9b6]) (logid:d4b118a0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:58,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-88a411b0]) (logid:6955814f) Enabled non-edge zones available for scan: 1 2025-06-29 04:18:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-88a411b0]) (logid:6955814f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:18:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-88a411b0]) (logid:6955814f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:18:58,257 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c449ca4a]) (logid:f40543be) HA health check task is running... 2025-06-29 04:18:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e4589049]) (logid:7d4222e2) Enabled non-edge zones available for scan: 1 2025-06-29 04:18:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e4589049]) (logid:7d4222e2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:18:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec4ef453]) (logid:885fff57) No inactive management server node found 2025-06-29 04:18:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec4ef453]) (logid:885fff57) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:18:59,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58b14744]) (logid:4c8cc7dd) No inactive management server node found 2025-06-29 04:18:59,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58b14744]) (logid:4c8cc7dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80795178]) (logid:e28e1426) No inactive management server node found 2025-06-29 04:19:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80795178]) (logid:e28e1426) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:02,258 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-528ee294]) (logid:a0f91f20) HA health check task is running... 2025-06-29 04:19:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d48ddb2f]) (logid:d9688459) No inactive management server node found 2025-06-29 04:19:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d48ddb2f]) (logid:d9688459) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5079c6f]) (logid:ee12204b) No inactive management server node found 2025-06-29 04:19:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5079c6f]) (logid:ee12204b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:05,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8556: Seq 7-8556: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:19:05,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8556 2025-06-29 04:19:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4f198f5]) (logid:553e7296) No inactive management server node found 2025-06-29 04:19:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4f198f5]) (logid:553e7296) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:06,259 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-97878deb]) (logid:05e7f814) HA health check task is running... 2025-06-29 04:19:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ec13c1f]) (logid:30c19896) No inactive management server node found 2025-06-29 04:19:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ec13c1f]) (logid:30c19896) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2303d51]) (logid:b9529d1e) No inactive management server node found 2025-06-29 04:19:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2303d51]) (logid:b9529d1e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:10,261 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-75f20791]) (logid:570b0df3) HA health check task is running... 2025-06-29 04:19:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cba8600d]) (logid:cd80b0b0) No inactive management server node found 2025-06-29 04:19:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cba8600d]) (logid:cd80b0b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a7edb84]) (logid:6fe73152) No inactive management server node found 2025-06-29 04:19:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a7edb84]) (logid:6fe73152) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bb463a6b]) (logid:7859652f) Found 0 routers to update status. 2025-06-29 04:19:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bb463a6b]) (logid:7859652f) Found 0 VPC's to update Redundant State. 2025-06-29 04:19:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-bb463a6b]) (logid:7859652f) Found 0 networks to update RvR status. 2025-06-29 04:19:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c085ec33]) (logid:470c41ea) No inactive management server node found 2025-06-29 04:19:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c085ec33]) (logid:470c41ea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:14,262 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-03b38c96]) (logid:b73704bc) HA health check task is running... 2025-06-29 04:19:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84c1c9ff]) (logid:e1b125a2) No inactive management server node found 2025-06-29 04:19:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84c1c9ff]) (logid:e1b125a2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:15,320 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8557: Seq 7-8557: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:19:15,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8557 2025-06-29 04:19:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-710631b2]) (logid:95b3f351) No inactive management server node found 2025-06-29 04:19:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-710631b2]) (logid:95b3f351) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e588bfa]) (logid:f9f6dc32) No inactive management server node found 2025-06-29 04:19:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e588bfa]) (logid:f9f6dc32) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:18,015 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-a3f22fcd]) (logid:7a7cf752) AutoScaling Monitor is running... 2025-06-29 04:19:18,080 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-f983b959]) (logid:25b8aa05) Starting to remove all volume_stats rows older than [Sat Jun 28 16:19:18 CEST 2025]. 2025-06-29 04:19:18,081 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-f983b959]) (logid:25b8aa05) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:19:18 CEST 2025]. 2025-06-29 04:19:18,263 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d390775f]) (logid:d4d8f5fb) HA health check task is running... 2025-06-29 04:19:18,425 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-b8a18ceb]) (logid:6efe4693) Starting to remove all vm_stats rows older than [Sat Jun 28 16:19:18 CEST 2025]. 2025-06-29 04:19:18,449 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-b8a18ceb]) (logid:6efe4693) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:19:18 CEST 2025]. 2025-06-29 04:19:19,267 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:19:19,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:19:19,270 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) Seq 2-4725683383995203768: Routed from 90520738888109 2025-06-29 04:19:19,313 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) Seq 1-4725683383995203768: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:19:19,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:19:19,315 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) Seq 2-3926857400090362054: Routed from 90520738888109 2025-06-29 04:19:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e62013b2]) (logid:0e56ddc0) No inactive management server node found 2025-06-29 04:19:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e62013b2]) (logid:0e56ddc0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:19,359 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3df44651]) (logid:6a1e3383) Seq 2-3926857400090362054: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:19:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00206b60]) (logid:fe1020bf) No inactive management server node found 2025-06-29 04:19:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00206b60]) (logid:fe1020bf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:21,437 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:19:21,441 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:19:21,441 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) Seq 2-4725683383995203769: Routed from 90520738888109 2025-06-29 04:19:21,497 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) Seq 1-4725683383995203769: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:19:21,538 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:19:21,539 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) Seq 2-3926857400090362055: Routed from 90520738888109 2025-06-29 04:19:21,630 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-5a12167a]) (logid:ea74bc87) Seq 2-3926857400090362055: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:19:22,061 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) StorageCollector is running... 2025-06-29 04:19:22,067 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:19:22,067 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:19:22,067 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:19:22,069 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-3938116399158788146: Routed from 90520738888109 2025-06-29 04:19:22,125 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 4-3938116399158788146: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:19:22,129 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:19:22,129 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:19:22,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:19:22,130 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-3926857400090362056: Routed from 90520738888109 2025-06-29 04:19:22,143 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-3926857400090362056: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:19:22,154 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:19:22,154 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:19:22,154 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:19:22,156 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-4725683383995203770: Routed from 90520738888109 2025-06-29 04:19:22,204 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 1-4725683383995203770: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:19:22,214 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:19:22,214 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:19:22,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:19:22,216 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-4725683383995203771: Routed from 90520738888109 2025-06-29 04:19:22,265 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-fa2f422d]) (logid:7a0e5b9d) HA health check task is running... 2025-06-29 04:19:22,301 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 1-4725683383995203771: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:19:22,302 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:19:22,303 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:19:22,303 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:19:22,304 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-3926857400090362057: Routed from 90520738888109 2025-06-29 04:19:22,349 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-35f10cd4]) (logid:21092aca) Seq 2-3926857400090362057: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:19:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8269310]) (logid:990697b4) No inactive management server node found 2025-06-29 04:19:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8269310]) (logid:990697b4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-173c678b]) (logid:2b4bc574) No inactive management server node found 2025-06-29 04:19:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-173c678b]) (logid:2b4bc574) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:25,320 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8558: Seq 7-8558: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:19:25,331 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8558 2025-06-29 04:19:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-792e054e]) (logid:a366ec0f) No inactive management server node found 2025-06-29 04:19:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-792e054e]) (logid:a366ec0f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:26,266 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-27a22c6d]) (logid:113fcc97) HA health check task is running... 2025-06-29 04:19:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eecbc1bb]) (logid:ae8f0bea) No inactive management server node found 2025-06-29 04:19:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eecbc1bb]) (logid:ae8f0bea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:19:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-061d4959]) (logid:a5613105) Enabled non-edge zones available for scan: 1 2025-06-29 04:19:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-061d4959]) (logid:a5613105) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:19:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-061d4959]) (logid:a5613105) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:19:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8850fb82]) (logid:b6a5d44a) Enabled non-edge zones available for scan: 1 2025-06-29 04:19:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8850fb82]) (logid:b6a5d44a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:19:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87e7abe5]) (logid:14fee7b1) No inactive management server node found 2025-06-29 04:19:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87e7abe5]) (logid:14fee7b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-250140d6]) (logid:99166e89) No inactive management server node found 2025-06-29 04:19:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-250140d6]) (logid:99166e89) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:30,267 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c5758b74]) (logid:038b2a86) HA health check task is running... 2025-06-29 04:19:30,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8559: Seq 7-8559: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:19:30,331 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8559 2025-06-29 04:19:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5bbe9c3]) (logid:dc9060ad) No inactive management server node found 2025-06-29 04:19:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5bbe9c3]) (logid:dc9060ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2af404]) (logid:b5ca9eb3) No inactive management server node found 2025-06-29 04:19:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2af404]) (logid:b5ca9eb3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:34,268 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a789b9fa]) (logid:6e44d0ae) HA health check task is running... 2025-06-29 04:19:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d13f9e8d]) (logid:292d7d29) No inactive management server node found 2025-06-29 04:19:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d13f9e8d]) (logid:292d7d29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad62a06]) (logid:d74e3a83) No inactive management server node found 2025-06-29 04:19:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad62a06]) (logid:d74e3a83) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2bb4931]) (logid:08f15b76) No inactive management server node found 2025-06-29 04:19:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2bb4931]) (logid:08f15b76) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:38,270 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-486091c9]) (logid:a4c22884) HA health check task is running... 2025-06-29 04:19:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cf3bc15]) (logid:33be72a8) No inactive management server node found 2025-06-29 04:19:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cf3bc15]) (logid:33be72a8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:40,320 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8560: Seq 7-8560: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:19:40,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8560 2025-06-29 04:19:40,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-090d0d1a]) (logid:667b7723) No inactive management server node found 2025-06-29 04:19:40,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-090d0d1a]) (logid:667b7723) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-171e3020]) (logid:9d6de05c) No inactive management server node found 2025-06-29 04:19:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-171e3020]) (logid:9d6de05c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:42,271 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-ef5dad86]) (logid:b9d80fb1) HA health check task is running... 2025-06-29 04:19:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-9282f060]) (logid:91d5140d) DbCollector is running... 2025-06-29 04:19:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e776936a]) (logid:2c62a2ca) Found 0 routers to update status. 2025-06-29 04:19:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e776936a]) (logid:2c62a2ca) Found 0 VPC's to update Redundant State. 2025-06-29 04:19:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e776936a]) (logid:2c62a2ca) Found 0 networks to update RvR status. 2025-06-29 04:19:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a281d667]) (logid:1d4156b4) No inactive management server node found 2025-06-29 04:19:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a281d667]) (logid:1d4156b4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) ManagementServerCollector is running... 2025-06-29 04:19:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) DB properties were already loaded 2025-06-29 04:19:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:19:44,060 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Execution of process [4066722] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:19:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Exit value of process [4066722] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:19:44,063 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Process [4066722] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:19:44,063 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) usage install: null 2025-06-29 04:19:44,063 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Sessions active: 0 2025-06-29 04:19:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:19:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066725] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:19:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:19:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066730] for command [/bin/bash -c uname -r ]. 2025-06-29 04:19:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:19:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066731] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:19:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) system memory from /proc: 4887515136 2025-06-29 04:19:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:19:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066736] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:19:44,086 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) free memory from /proc: 295723008 2025-06-29 04:19:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:19:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066741] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:19:44,100 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) used memory from /proc: 1071228 2025-06-29 04:19:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:19:44,105 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066742] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:19:44,105 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:19:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066743] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:19:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:19:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066744] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:19:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:19:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066751] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:19:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:19:44,133 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Successfully executed process [4066752] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail buffersdirect.count, 93 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail buffersdirect.used, 321663 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail buffersmapped.capacity, 0 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail buffersmapped.count, 0 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail buffersmapped.used, 0 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:19:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail jvmuptime, 2643839 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memoryheap.committed, 620232704 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memoryheap.init, 77594624 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memoryheap.usage, 0.18250517935518445 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorynon-heap.committed, 196739072 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorynon-heap.max, -1 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorynon-heap.usage, -1.94360664E8 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorynon-heap.used, 194360664 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11432263896294322 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21961229879694738 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.Metaspace.usage, 0.9926544692095588 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.PS-Eden-Space.usage, 0.25465993471858234 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorytotal.committed, 816971776 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorytotal.init, 85262336 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorytotal.max, 1908932607 2025-06-29 04:19:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail memorytotal.used, 542922224 2025-06-29 04:19:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail threadsdeadlocks, [] 2025-06-29 04:19:44,137 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail threadsnew.count, 0 2025-06-29 04:19:44,138 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:19:44,139 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-ca1995fb]) (logid:0fb7fc91) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:19:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":272471840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2643766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":542121528,"totalCommitted":816971776,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":348390088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":295723008,"systemMemoryUsed":1071228,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.24,0.44],"systemCyclesUsage":[66412914,53833396,4589622121],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:19:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 87, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:19:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":272471840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2643766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":542121528,"totalCommitted":816971776,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":348390088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":295723008,"systemMemoryUsed":1071228,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.24,0.44],"systemCyclesUsage":[66412914,53833396,4589622121],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:19:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:19:44,148 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-21:[ctx-082ab7a8]) (logid:a2fd45ee) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:19:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":272471840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2643766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":542121528,"totalCommitted":816971776,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":348390088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":295723008,"systemMemoryUsed":1071228,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.24,0.44],"systemCyclesUsage":[66412914,53833396,4589622121],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:19:44,148 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 04:19:44,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 87, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:19:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":272471840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2643766,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":542121528,"totalCommitted":816971776,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":348390088,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":295723008,"systemMemoryUsed":1071228,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.24,0.44],"systemCyclesUsage":[66412914,53833396,4589622121],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:19:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e39413]) (logid:5c0a52dd) No inactive management server node found 2025-06-29 04:19:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56e39413]) (logid:5c0a52dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:19:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:19:46,272 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-21aae6d2]) (logid:92bbf5f2) HA health check task is running... 2025-06-29 04:19:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-337f8ada]) (logid:c4bae6de) No inactive management server node found 2025-06-29 04:19:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-337f8ada]) (logid:c4bae6de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:19:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:19:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:19:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:19:47,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:19:47,091 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:19:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:19:47,094 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:19:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78cdf5eb]) (logid:a44e8b12) No inactive management server node found 2025-06-29 04:19:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78cdf5eb]) (logid:a44e8b12) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3a41f6c5]) (logid:260ad4db) VM scheduler.poll is being called at 2025-06-29T02:20:00+0000 2025-06-29 04:19:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3a41f6c5]) (logid:260ad4db) Got 0 scheduled jobs to be executed at 2025-06-29T02:20:00+0000 2025-06-29 04:19:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3a41f6c5]) (logid:260ad4db) Cleaned up 0 VM scheduled job entries 2025-06-29 04:19:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-fcdcff68]) (logid:4e209dea) VM Operation Thread Running 2025-06-29 04:19:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-57952560]) (logid:7d554205) ClusterDRS.poll is being called at 2025-06-29T02:20:00+0000 2025-06-29 04:19:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-57952560]) (logid:7d554205) Removed 0 old drs migration plans 2025-06-29 04:19:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11810008]) (logid:8de69a57) No inactive management server node found 2025-06-29 04:19:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11810008]) (logid:8de69a57) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:50,273 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-be4e5704]) (logid:496e2f0c) HA health check task is running... 2025-06-29 04:19:50,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8562: Seq 7-8562: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:19:50,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8562 2025-06-29 04:19:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b9b4c41]) (logid:514d1053) No inactive management server node found 2025-06-29 04:19:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b9b4c41]) (logid:514d1053) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9514b175]) (logid:24b3c239) No inactive management server node found 2025-06-29 04:19:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9514b175]) (logid:24b3c239) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d639ed9]) (logid:a06dcd5c) No inactive management server node found 2025-06-29 04:19:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d639ed9]) (logid:a06dcd5c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:54,275 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1a5bb667]) (logid:6400f118) HA health check task is running... 2025-06-29 04:19:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5faa0068]) (logid:f4bd1854) No inactive management server node found 2025-06-29 04:19:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5faa0068]) (logid:f4bd1854) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea91d5a2]) (logid:e41676da) No inactive management server node found 2025-06-29 04:19:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea91d5a2]) (logid:e41676da) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c754a24]) (logid:a8f24317) Enabled non-edge zones available for scan: 1 2025-06-29 04:19:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1c754a24]) (logid:a8f24317) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:19:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1c754a24]) (logid:a8f24317) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:19:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f22e191b]) (logid:3381a78a) Enabled non-edge zones available for scan: 1 2025-06-29 04:19:58,276 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9214a6bf]) (logid:fa588816) HA health check task is running... 2025-06-29 04:19:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f22e191b]) (logid:3381a78a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:19:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cf2af2a]) (logid:b2b19e3c) No inactive management server node found 2025-06-29 04:19:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cf2af2a]) (logid:b2b19e3c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:19:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-704a11b1]) (logid:1768c768) No inactive management server node found 2025-06-29 04:19:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-704a11b1]) (logid:1768c768) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:00,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8563: Seq 7-8563: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:00,332 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8563 2025-06-29 04:20:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-109f55e9]) (logid:8930585d) No inactive management server node found 2025-06-29 04:20:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-109f55e9]) (logid:8930585d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:02,278 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-727063e1]) (logid:f9573aac) HA health check task is running... 2025-06-29 04:20:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bd1131d]) (logid:4642458a) No inactive management server node found 2025-06-29 04:20:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bd1131d]) (logid:4642458a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3547af7d]) (logid:c794b467) No inactive management server node found 2025-06-29 04:20:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3547af7d]) (logid:c794b467) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01206159]) (logid:dbc66406) No inactive management server node found 2025-06-29 04:20:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01206159]) (logid:dbc66406) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:06,279 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9fff6570]) (logid:6b063c44) HA health check task is running... 2025-06-29 04:20:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1746bef5]) (logid:6ad9c771) No inactive management server node found 2025-06-29 04:20:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1746bef5]) (logid:6ad9c771) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23748d65]) (logid:5fd4edef) No inactive management server node found 2025-06-29 04:20:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-23748d65]) (logid:5fd4edef) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:10,280 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cd7db9bd]) (logid:4165e285) HA health check task is running... 2025-06-29 04:20:10,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8564: Seq 7-8564: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:10,331 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8564 2025-06-29 04:20:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-021a7177]) (logid:b5dc1375) No inactive management server node found 2025-06-29 04:20:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-021a7177]) (logid:b5dc1375) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d3451af]) (logid:b46c5838) No inactive management server node found 2025-06-29 04:20:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d3451af]) (logid:b46c5838) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-efe27428]) (logid:cbc78734) Found 0 routers to update status. 2025-06-29 04:20:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-efe27428]) (logid:cbc78734) Found 0 VPC's to update Redundant State. 2025-06-29 04:20:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-efe27428]) (logid:cbc78734) Found 0 networks to update RvR status. 2025-06-29 04:20:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91a24140]) (logid:90f440db) No inactive management server node found 2025-06-29 04:20:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91a24140]) (logid:90f440db) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:14,281 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-eec838a1]) (logid:f0472440) HA health check task is running... 2025-06-29 04:20:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f70bcc8]) (logid:1fdabc40) No inactive management server node found 2025-06-29 04:20:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f70bcc8]) (logid:1fdabc40) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:15,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8565: Seq 7-8565: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:15,331 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8565 2025-06-29 04:20:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e566923e]) (logid:bf03ff9d) No inactive management server node found 2025-06-29 04:20:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e566923e]) (logid:bf03ff9d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:17,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8cc2042]) (logid:cbadebc2) No inactive management server node found 2025-06-29 04:20:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8cc2042]) (logid:cbadebc2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:18,017 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-8c023b01]) (logid:355565fc) AutoScaling Monitor is running... 2025-06-29 04:20:18,083 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-863ad3f3]) (logid:2bcf00bd) Starting to remove all volume_stats rows older than [Sat Jun 28 16:20:18 CEST 2025]. 2025-06-29 04:20:18,083 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-863ad3f3]) (logid:2bcf00bd) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:20:18 CEST 2025]. 2025-06-29 04:20:18,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-293b6c7c]) (logid:4bda362f) Resetting hosts suitable for reconnect 2025-06-29 04:20:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-293b6c7c]) (logid:4bda362f) Completed resetting hosts suitable for reconnect 2025-06-29 04:20:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-293b6c7c]) (logid:4bda362f) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:20:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-293b6c7c]) (logid:4bda362f) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:20:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-293b6c7c]) (logid:4bda362f) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:20:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-293b6c7c]) (logid:4bda362f) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:20:18,283 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-67b9a050]) (logid:1437b96f) HA health check task is running... 2025-06-29 04:20:18,451 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a478610e]) (logid:b6c28816) Starting to remove all vm_stats rows older than [Sat Jun 28 16:20:18 CEST 2025]. 2025-06-29 04:20:18,476 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a478610e]) (logid:b6c28816) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:20:18 CEST 2025]. 2025-06-29 04:20:19,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8669b3d0]) (logid:a5a7859d) No inactive management server node found 2025-06-29 04:20:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8669b3d0]) (logid:a5a7859d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:19,361 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:20:19,363 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:20:19,364 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) Seq 2-4725683383995203772: Routed from 90520738888109 2025-06-29 04:20:19,408 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) Seq 1-4725683383995203772: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:20:19,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:20:19,410 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) Seq 2-3926857400090362058: Routed from 90520738888109 2025-06-29 04:20:19,454 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-eff204e2]) (logid:1c7b4e4d) Seq 2-3926857400090362058: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:20:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba20e005]) (logid:95fc86e8) No inactive management server node found 2025-06-29 04:20:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba20e005]) (logid:95fc86e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:21,691 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:20:21,695 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:20:21,696 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) Seq 2-4725683383995203773: Routed from 90520738888109 2025-06-29 04:20:21,752 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) Seq 1-4725683383995203773: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:20:21,793 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:20:21,794 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) Seq 2-3926857400090362059: Routed from 90520738888109 2025-06-29 04:20:21,885 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-e1122c51]) (logid:2813bfdf) Seq 2-3926857400090362059: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:20:22,284 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f3b79cfc]) (logid:e4481021) HA health check task is running... 2025-06-29 04:20:22,350 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) StorageCollector is running... 2025-06-29 04:20:22,359 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:20:22,359 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:20:22,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:20:22,360 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3938116399158788147: Routed from 90520738888109 2025-06-29 04:20:22,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f83c92e]) (logid:619dd1f0) No inactive management server node found 2025-06-29 04:20:22,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f83c92e]) (logid:619dd1f0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:22,420 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 4-3938116399158788147: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:20:22,424 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:20:22,424 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:20:22,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:20:22,426 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3926857400090362060: Routed from 90520738888109 2025-06-29 04:20:22,438 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3926857400090362060: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:20:22,449 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:20:22,449 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:20:22,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:20:22,450 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3926857400090362061: Routed from 90520738888109 2025-06-29 04:20:22,544 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3926857400090362061: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:20:22,554 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:20:22,554 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:20:22,554 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:20:22,555 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-4725683383995203774: Routed from 90520738888109 2025-06-29 04:20:22,599 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 1-4725683383995203774: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:20:22,601 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:20:22,601 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:20:22,601 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:20:22,602 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3926857400090362062: Routed from 90520738888109 2025-06-29 04:20:22,647 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a490c542]) (logid:d66db83e) Seq 2-3926857400090362062: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:20:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82509e4c]) (logid:5b4fe8e1) No inactive management server node found 2025-06-29 04:20:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82509e4c]) (logid:5b4fe8e1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:25,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8566: Seq 7-8566: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:25,343 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8566 2025-06-29 04:20:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-363728a9]) (logid:08d761c8) No inactive management server node found 2025-06-29 04:20:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-363728a9]) (logid:08d761c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:26,286 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2f64b221]) (logid:49a64f12) HA health check task is running... 2025-06-29 04:20:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cbec8d9]) (logid:7d7a2a9e) No inactive management server node found 2025-06-29 04:20:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cbec8d9]) (logid:7d7a2a9e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-06d14ded]) (logid:eca3bbc2) Enabled non-edge zones available for scan: 1 2025-06-29 04:20:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-06d14ded]) (logid:eca3bbc2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:20:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-06d14ded]) (logid:eca3bbc2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:20:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c3a59e26]) (logid:0afd56c7) Enabled non-edge zones available for scan: 1 2025-06-29 04:20:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c3a59e26]) (logid:0afd56c7) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:20:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65baec18]) (logid:29365184) No inactive management server node found 2025-06-29 04:20:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65baec18]) (logid:29365184) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3afaa158]) (logid:f5056101) No inactive management server node found 2025-06-29 04:20:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3afaa158]) (logid:f5056101) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:30,287 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ca795eb9]) (logid:cc1e0818) HA health check task is running... 2025-06-29 04:20:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5160d07]) (logid:c1a0a45b) No inactive management server node found 2025-06-29 04:20:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5160d07]) (logid:c1a0a45b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fab8eb8]) (logid:5df166e6) No inactive management server node found 2025-06-29 04:20:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fab8eb8]) (logid:5df166e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:34,288 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-82bcfe9b]) (logid:a7098622) HA health check task is running... 2025-06-29 04:20:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29a2de4a]) (logid:cf7c913b) No inactive management server node found 2025-06-29 04:20:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29a2de4a]) (logid:cf7c913b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:35,322 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8567: Seq 7-8567: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:35,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8567 2025-06-29 04:20:35,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89cf287f]) (logid:d480cdbd) No inactive management server node found 2025-06-29 04:20:35,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89cf287f]) (logid:d480cdbd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5388c25]) (logid:30bfe94b) No inactive management server node found 2025-06-29 04:20:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5388c25]) (logid:30bfe94b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:38,290 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-99365308]) (logid:ea98344b) HA health check task is running... 2025-06-29 04:20:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfad3a1d]) (logid:b7afa1f3) No inactive management server node found 2025-06-29 04:20:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfad3a1d]) (logid:b7afa1f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50dd5eb7]) (logid:24504971) No inactive management server node found 2025-06-29 04:20:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50dd5eb7]) (logid:24504971) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47015388]) (logid:a1149bb2) No inactive management server node found 2025-06-29 04:20:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47015388]) (logid:a1149bb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:42,291 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-d47320a9]) (logid:b75c86ff) HA health check task is running... 2025-06-29 04:20:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-11964398]) (logid:dbcc5789) DbCollector is running... 2025-06-29 04:20:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f3a52a1a]) (logid:11ef11eb) Found 0 routers to update status. 2025-06-29 04:20:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f3a52a1a]) (logid:11ef11eb) Found 0 VPC's to update Redundant State. 2025-06-29 04:20:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f3a52a1a]) (logid:11ef11eb) Found 0 networks to update RvR status. 2025-06-29 04:20:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8898eee5]) (logid:2de4cad1) No inactive management server node found 2025-06-29 04:20:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8898eee5]) (logid:2de4cad1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) ManagementServerCollector is running... 2025-06-29 04:20:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) DB properties were already loaded 2025-06-29 04:20:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:20:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Execution of process [4066766] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:20:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Exit value of process [4066766] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:20:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Process [4066766] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:20:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) usage install: null 2025-06-29 04:20:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Sessions active: 0 2025-06-29 04:20:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:20:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066769] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:20:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:20:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066774] for command [/bin/bash -c uname -r ]. 2025-06-29 04:20:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:20:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066775] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:20:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) system memory from /proc: 4887515136 2025-06-29 04:20:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:20:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066780] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:20:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) free memory from /proc: 292921344 2025-06-29 04:20:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:20:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066785] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:20:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) used memory from /proc: 1071484 2025-06-29 04:20:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:20:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066786] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:20:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:20:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066787] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:20:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:20:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066788] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:20:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:20:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066795] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:20:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:20:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Successfully executed process [4066796] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:20:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:20:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail buffersdirect.count, 93 2025-06-29 04:20:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail buffersdirect.used, 321663 2025-06-29 04:20:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail buffersmapped.capacity, 0 2025-06-29 04:20:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail buffersmapped.count, 0 2025-06-29 04:20:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail buffersmapped.used, 0 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail jvmuptime, 2703829 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memoryheap.committed, 620232704 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memoryheap.init, 77594624 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memoryheap.usage, 0.19409046000224225 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorynon-heap.committed, 196870144 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorynon-heap.max, -1 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorynon-heap.usage, -1.94490944E8 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorynon-heap.used, 194490944 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11457569648093842 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22027880994434632 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.Metaspace.usage, 0.9927783777573529 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.PS-Eden-Space.usage, 0.2879309601169701 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorytotal.committed, 817102848 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorytotal.init, 85262336 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorytotal.max, 1908932607 2025-06-29 04:20:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail memorytotal.used, 564996552 2025-06-29 04:20:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail threadsdeadlocks, [] 2025-06-29 04:20:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail threadsnew.count, 0 2025-06-29 04:20:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:20:44,130 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-b51ecc9a]) (logid:590cc957) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:20:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":250427840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2703759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":564295808,"totalCommitted":817102848,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":370505608,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292921344,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.19,0.41],"systemCyclesUsage":[66413243,53833698,4589645240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:20:44,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 89, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:20:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":250427840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2703759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":564295808,"totalCommitted":817102848,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":370505608,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292921344,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.19,0.41],"systemCyclesUsage":[66413243,53833698,4589645240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:20:44,131 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:20:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-21:[ctx-a86c1a46]) (logid:dcae596a) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:20:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":250427840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2703759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":564295808,"totalCommitted":817102848,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":370505608,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292921344,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.19,0.41],"systemCyclesUsage":[66413243,53833698,4589645240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:20:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:20:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 89, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:20:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":250427840,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2703759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":564295808,"totalCommitted":817102848,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":370505608,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":292921344,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.19,0.41],"systemCyclesUsage":[66413243,53833698,4589645240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:20:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73ee1402]) (logid:54be3a6c) No inactive management server node found 2025-06-29 04:20:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73ee1402]) (logid:54be3a6c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:45,323 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8568: Seq 7-8568: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:45,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8568 2025-06-29 04:20:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:20:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:20:46,293 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-248c3a8c]) (logid:4efd3522) HA health check task is running... 2025-06-29 04:20:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e0f5e8f]) (logid:f981a441) No inactive management server node found 2025-06-29 04:20:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e0f5e8f]) (logid:f981a441) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:20:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:20:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:20:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:20:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:20:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:20:47,074 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:20:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:20:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6831ff30]) (logid:97e20ee9) No inactive management server node found 2025-06-29 04:20:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6831ff30]) (logid:97e20ee9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a5152371]) (logid:ed9a5dfa) VM scheduler.poll is being called at 2025-06-29T02:21:00+0000 2025-06-29 04:20:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a5152371]) (logid:ed9a5dfa) Got 0 scheduled jobs to be executed at 2025-06-29T02:21:00+0000 2025-06-29 04:20:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a5152371]) (logid:ed9a5dfa) Cleaned up 0 VM scheduled job entries 2025-06-29 04:20:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-e5bd5fcb]) (logid:d9e1f40c) VM Operation Thread Running 2025-06-29 04:20:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-c60ff947]) (logid:ff3d5ff5) ClusterDRS.poll is being called at 2025-06-29T02:21:00+0000 2025-06-29 04:20:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-c60ff947]) (logid:ff3d5ff5) Removed 0 old drs migration plans 2025-06-29 04:20:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c6602a7]) (logid:5675964c) No inactive management server node found 2025-06-29 04:20:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c6602a7]) (logid:5675964c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:50,294 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-15183966]) (logid:7b6ab171) HA health check task is running... 2025-06-29 04:20:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a567cfc]) (logid:2cfe8e0b) No inactive management server node found 2025-06-29 04:20:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a567cfc]) (logid:2cfe8e0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc508296]) (logid:706f92c6) No inactive management server node found 2025-06-29 04:20:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc508296]) (logid:706f92c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1925bb0d]) (logid:cfea5667) No inactive management server node found 2025-06-29 04:20:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1925bb0d]) (logid:cfea5667) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:54,296 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fe3b8c09]) (logid:334f02bc) HA health check task is running... 2025-06-29 04:20:55,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8570: Seq 7-8570: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:20:55,334 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8570 2025-06-29 04:20:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4b60d03]) (logid:5021cda4) No inactive management server node found 2025-06-29 04:20:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4b60d03]) (logid:5021cda4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f8dad34]) (logid:e6b44331) No inactive management server node found 2025-06-29 04:20:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f8dad34]) (logid:e6b44331) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ede1a650]) (logid:ef890089) Enabled non-edge zones available for scan: 1 2025-06-29 04:20:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ede1a650]) (logid:ef890089) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:20:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ede1a650]) (logid:ef890089) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:20:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ed3cb389]) (logid:25692b35) Enabled non-edge zones available for scan: 1 2025-06-29 04:20:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ed3cb389]) (logid:25692b35) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:20:58,297 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a496f767]) (logid:832a1d8c) HA health check task is running... 2025-06-29 04:20:58,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b61a1fcb]) (logid:44ade67c) No inactive management server node found 2025-06-29 04:20:58,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b61a1fcb]) (logid:44ade67c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:20:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d9a1d29]) (logid:0423d0e4) No inactive management server node found 2025-06-29 04:20:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d9a1d29]) (logid:0423d0e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08beddc6]) (logid:4d73105b) No inactive management server node found 2025-06-29 04:21:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08beddc6]) (logid:4d73105b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:02,299 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0ae685ce]) (logid:0f76b7d2) HA health check task is running... 2025-06-29 04:21:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fd81bdd]) (logid:505d845e) No inactive management server node found 2025-06-29 04:21:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fd81bdd]) (logid:505d845e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ee47295]) (logid:52dd92d5) No inactive management server node found 2025-06-29 04:21:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ee47295]) (logid:52dd92d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:05,325 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8571: Seq 7-8571: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:21:05,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8571 2025-06-29 04:21:05,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6763b1e8]) (logid:e0071657) No inactive management server node found 2025-06-29 04:21:05,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6763b1e8]) (logid:e0071657) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:06,300 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ca61c207]) (logid:289d3343) HA health check task is running... 2025-06-29 04:21:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a01336be]) (logid:ea7853fe) No inactive management server node found 2025-06-29 04:21:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a01336be]) (logid:ea7853fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b391d80]) (logid:80ecd8c3) No inactive management server node found 2025-06-29 04:21:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b391d80]) (logid:80ecd8c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:10,301 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-18ecb05c]) (logid:45098956) HA health check task is running... 2025-06-29 04:21:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a90c6013]) (logid:9d0383c0) No inactive management server node found 2025-06-29 04:21:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a90c6013]) (logid:9d0383c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6c069cf]) (logid:2a713f19) No inactive management server node found 2025-06-29 04:21:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e6c069cf]) (logid:2a713f19) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad7d6fd2]) (logid:2ac45718) Found 0 routers to update status. 2025-06-29 04:21:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad7d6fd2]) (logid:2ac45718) Found 0 VPC's to update Redundant State. 2025-06-29 04:21:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ad7d6fd2]) (logid:2ac45718) Found 0 networks to update RvR status. 2025-06-29 04:21:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3eaf2f0]) (logid:e9e505e0) No inactive management server node found 2025-06-29 04:21:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3eaf2f0]) (logid:e9e505e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:14,303 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f2caf77a]) (logid:2c324986) HA health check task is running... 2025-06-29 04:21:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc78d615]) (logid:65242897) No inactive management server node found 2025-06-29 04:21:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc78d615]) (logid:65242897) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:15,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8572: Seq 7-8572: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:21:15,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8572 2025-06-29 04:21:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f930c61]) (logid:7b0c192b) No inactive management server node found 2025-06-29 04:21:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f930c61]) (logid:7b0c192b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e291048]) (logid:8a9c91cb) No inactive management server node found 2025-06-29 04:21:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e291048]) (logid:8a9c91cb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:18,019 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-9809ee65]) (logid:6cdb6aaf) AutoScaling Monitor is running... 2025-06-29 04:21:18,085 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-ce16a235]) (logid:cbe2f4cd) Starting to remove all volume_stats rows older than [Sat Jun 28 16:21:18 CEST 2025]. 2025-06-29 04:21:18,086 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-ce16a235]) (logid:cbe2f4cd) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:21:18 CEST 2025]. 2025-06-29 04:21:18,305 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f884222c]) (logid:31d7bc1f) HA health check task is running... 2025-06-29 04:21:18,478 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-27dffd55]) (logid:ca3ad7b8) Starting to remove all vm_stats rows older than [Sat Jun 28 16:21:18 CEST 2025]. 2025-06-29 04:21:18,516 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-27dffd55]) (logid:ca3ad7b8) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:21:18 CEST 2025]. 2025-06-29 04:21:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c1398db]) (logid:732f913a) No inactive management server node found 2025-06-29 04:21:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c1398db]) (logid:732f913a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:21:19,457 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:21:19,458 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:21:19,459 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) Seq 2-4725683383995203775: Routed from 90520738888109 2025-06-29 04:21:19,502 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) Seq 1-4725683383995203775: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:21:19,504 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:21:19,505 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) Seq 2-3926857400090362063: Routed from 90520738888109 2025-06-29 04:21:19,549 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ce8c1297]) (logid:14eadd50) Seq 2-3926857400090362063: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:21:20,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8573: Seq 7-8573: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:21:20,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8573 2025-06-29 04:21:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87e3326a]) (logid:962c4b39) No inactive management server node found 2025-06-29 04:21:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87e3326a]) (logid:962c4b39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:21,944 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:21:21,947 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:21:21,948 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) Seq 2-4725683383995203776: Routed from 90520738888109 2025-06-29 04:21:22,015 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) Seq 1-4725683383995203776: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:21:22,051 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:21:22,052 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) Seq 2-3926857400090362064: Routed from 90520738888109 2025-06-29 04:21:22,153 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-10e642f5]) (logid:68b65b91) Seq 2-3926857400090362064: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:21:22,306 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e440007c]) (logid:136e9539) HA health check task is running... 2025-06-29 04:21:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfa01712]) (logid:a537e058) No inactive management server node found 2025-06-29 04:21:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfa01712]) (logid:a537e058) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:21:22,648 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) StorageCollector is running... 2025-06-29 04:21:22,654 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:21:22,654 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:21:22,654 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:21:22,655 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-3938116399158788148: Routed from 90520738888109 2025-06-29 04:21:22,733 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 4-3938116399158788148: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:21:22,737 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:21:22,737 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:21:22,737 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:21:22,738 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-3926857400090362065: Routed from 90520738888109 2025-06-29 04:21:22,749 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-3926857400090362065: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:21:22,760 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:21:22,760 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:21:22,760 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:21:22,761 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-4725683383995203777: Routed from 90520738888109 2025-06-29 04:21:22,808 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 1-4725683383995203777: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:21:22,820 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:21:22,820 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:21:22,820 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:21:22,821 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-4725683383995203778: Routed from 90520738888109 2025-06-29 04:21:22,904 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 1-4725683383995203778: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:21:22,906 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:21:22,906 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:21:22,906 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:21:22,907 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-3926857400090362066: Routed from 90520738888109 2025-06-29 04:21:22,951 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-b028c03f]) (logid:01dfeb8b) Seq 2-3926857400090362066: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:21:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c04c1941]) (logid:dd6692ef) No inactive management server node found 2025-06-29 04:21:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c04c1941]) (logid:dd6692ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:21:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d31acf7d]) (logid:364f128a) No inactive management server node found 2025-06-29 04:21:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d31acf7d]) (logid:364f128a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:21:26,307 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c10bbb0c]) (logid:3b8252f1) HA health check task is running... 2025-06-29 04:21:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a52936c7]) (logid:ba2b926d) No inactive management server node found 2025-06-29 04:21:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a52936c7]) (logid:ba2b926d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-33535f28]) (logid:75bcd7c8) Enabled non-edge zones available for scan: 1 2025-06-29 04:21:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-33535f28]) (logid:75bcd7c8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:21:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-33535f28]) (logid:75bcd7c8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:21:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-91b7b009]) (logid:4597db16) Enabled non-edge zones available for scan: 1 2025-06-29 04:21:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-91b7b009]) (logid:4597db16) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:21:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd24bb44]) (logid:8b550bfa) No inactive management server node found 2025-06-29 04:21:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd24bb44]) (logid:8b550bfa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04dbe7d7]) (logid:2cfc217d) No inactive management server node found 2025-06-29 04:21:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04dbe7d7]) (logid:2cfc217d) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:30,308 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-30ed4fbc]) (logid:c4d3420e) HA health check task is running... 2025-06-29 04:21:30,325 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8574: Seq 7-8574: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:21:30,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8574 2025-06-29 04:21:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-896bce41]) (logid:15d18c97) No inactive management server node found 2025-06-29 04:21:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-896bce41]) (logid:15d18c97) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c300dcaa]) (logid:818380ef) No inactive management server node found 2025-06-29 04:21:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c300dcaa]) (logid:818380ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:34,310 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-26d49354]) (logid:82622421) HA health check task is running... 2025-06-29 04:21:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2254b782]) (logid:04b9201f) No inactive management server node found 2025-06-29 04:21:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2254b782]) (logid:04b9201f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:35,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01f4fed1]) (logid:07211740) No inactive management server node found 2025-06-29 04:21:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01f4fed1]) (logid:07211740) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de4b62ba]) (logid:6188d3e2) No inactive management server node found 2025-06-29 04:21:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de4b62ba]) (logid:6188d3e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:38,311 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a869e0cd]) (logid:d8d96808) HA health check task is running... 2025-06-29 04:21:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e783acb]) (logid:27bd0f34) No inactive management server node found 2025-06-29 04:21:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e783acb]) (logid:27bd0f34) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:40,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8575: Seq 7-8575: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:21:40,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8575 2025-06-29 04:21:40,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65e1246c]) (logid:43297d47) No inactive management server node found 2025-06-29 04:21:40,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65e1246c]) (logid:43297d47) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef3ec059]) (logid:443cd39b) No inactive management server node found 2025-06-29 04:21:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef3ec059]) (logid:443cd39b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:42,313 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5771af93]) (logid:4990a5f0) HA health check task is running... 2025-06-29 04:21:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-d307df17]) (logid:6868d3f7) DbCollector is running... 2025-06-29 04:21:43,093 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-ef981a3f]) (logid:52073062) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:21:43,094 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-ef981a3f]) (logid:52073062) Snapshot scheduler is being called at [2025-06-29T02:21:43+0000]. 2025-06-29 04:21:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-ef981a3f]) (logid:52073062) There are [0] scheduled snapshots to be executed at [2025-06-29T02:21:43+0000]. 2025-06-29 04:21:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-9e919471]) (logid:a0291964) Found 0 running routers. 2025-06-29 04:21:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d32bc40]) (logid:a1b9640b) Found 0 routers to update status. 2025-06-29 04:21:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d32bc40]) (logid:a1b9640b) Found 0 VPC's to update Redundant State. 2025-06-29 04:21:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d32bc40]) (logid:a1b9640b) Found 0 networks to update RvR status. 2025-06-29 04:21:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-3c3d137d]) (logid:70bd327e) External devices are not used. Skipping external device usage collection 2025-06-29 04:21:43,131 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-93ca3538]) (logid:4bbc9e54) Started resource counters recalculation periodic task. 2025-06-29 04:21:43,134 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-93ca3538]) (logid:4bbc9e54) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:21:43,306 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-50cf01de]) (logid:a79d71c6) Backup backup.poll is being called at 2025-06-29T02:21:43+0000 2025-06-29 04:21:43,357 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b60ffde1]) (logid:761d30d3) No inactive management server node found 2025-06-29 04:21:43,357 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b60ffde1]) (logid:761d30d3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:43,525 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-93ca3538]) (logid:4bbc9e54) Finished resource counters recalculation periodic task. 2025-06-29 04:21:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) ManagementServerCollector is running... 2025-06-29 04:21:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) DB properties were already loaded 2025-06-29 04:21:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:21:44,059 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Execution of process [4066801] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:21:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Exit value of process [4066801] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:21:44,059 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Process [4066801] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:21:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) usage install: null 2025-06-29 04:21:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Sessions active: 0 2025-06-29 04:21:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:21:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066804] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:21:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:21:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066809] for command [/bin/bash -c uname -r ]. 2025-06-29 04:21:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:21:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066810] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:21:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) system memory from /proc: 4887515136 2025-06-29 04:21:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:21:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066815] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:21:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) free memory from /proc: 297000960 2025-06-29 04:21:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:21:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066820] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:21:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) used memory from /proc: 1071484 2025-06-29 04:21:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:21:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066821] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:21:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:21:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066822] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:21:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:21:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066823] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:21:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:21:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066830] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:21:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:21:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Successfully executed process [4066831] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail buffersdirect.count, 93 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail buffersdirect.used, 321663 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail buffersmapped.capacity, 0 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail buffersmapped.count, 0 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail buffersmapped.used, 0 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail jvmuptime, 2763832 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memoryheap.committed, 620232704 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memoryheap.init, 77594624 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memoryheap.usage, 0.21082543737447645 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorynon-heap.committed, 196935680 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorynon-heap.max, -1 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorynon-heap.usage, -1.94623328E8 2025-06-29 04:21:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorynon-heap.used, 194623328 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11462880731804852 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22116297863831771 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.Metaspace.usage, 0.9929017693014706 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.PS-Eden-Space.usage, 0.33636661402182283 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorytotal.committed, 817168384 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorytotal.init, 85262336 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorytotal.max, 1908932607 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail memorytotal.used, 597258752 2025-06-29 04:21:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail threadsdeadlocks, [] 2025-06-29 04:21:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail threadsnew.count, 0 2025-06-29 04:21:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:21:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-28aeb787]) (logid:5e4797f2) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:21:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":218367720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2763762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":596484624,"totalCommitted":817168384,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402451552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297000960,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.16,0.39],"systemCyclesUsage":[66413586,53834006,4589668342],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:21:44,133 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 91, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:21:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":218367720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2763762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":596484624,"totalCommitted":817168384,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402451552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297000960,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.16,0.39],"systemCyclesUsage":[66413586,53834006,4589668342],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:21:44,133 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:21:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-21:[ctx-506f8975]) (logid:65670447) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:21:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":218367720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2763762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":596484624,"totalCommitted":817168384,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402451552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297000960,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.16,0.39],"systemCyclesUsage":[66413586,53834006,4589668342],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:21:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:21:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 91, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:21:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":218367720,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2763762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":596484624,"totalCommitted":817168384,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402451552,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297000960,"systemMemoryUsed":1071484,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.16,0.39],"systemCyclesUsage":[66413586,53834006,4589668342],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:21:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c99296d5]) (logid:61559075) No inactive management server node found 2025-06-29 04:21:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c99296d5]) (logid:61559075) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:21:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:21:46,314 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e05c57e4]) (logid:f4437f49) HA health check task is running... 2025-06-29 04:21:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71328f21]) (logid:37c644c0) No inactive management server node found 2025-06-29 04:21:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71328f21]) (logid:37c644c0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:46,646 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:21:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:21:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:21:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:21:47,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:21:47,094 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:21:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:21:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:21:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbc5e9ea]) (logid:d65d0f7a) No inactive management server node found 2025-06-29 04:21:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fbc5e9ea]) (logid:d65d0f7a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:48,050 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-6:[ctx-b9b5c1ab]) (logid:8315d74b) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:21:48,092 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dc7131d1]) (logid:2539c716) Resetting hosts suitable for reconnect 2025-06-29 04:21:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dc7131d1]) (logid:2539c716) Completed resetting hosts suitable for reconnect 2025-06-29 04:21:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dc7131d1]) (logid:2539c716) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:21:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dc7131d1]) (logid:2539c716) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:21:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dc7131d1]) (logid:2539c716) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:21:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-dc7131d1]) (logid:2539c716) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:21:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-39d38707]) (logid:072e4892) VM scheduler.poll is being called at 2025-06-29T02:22:00+0000 2025-06-29 04:21:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-39d38707]) (logid:072e4892) Got 0 scheduled jobs to be executed at 2025-06-29T02:22:00+0000 2025-06-29 04:21:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-39d38707]) (logid:072e4892) Cleaned up 0 VM scheduled job entries 2025-06-29 04:21:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-7ee5550d]) (logid:bd783831) VM Operation Thread Running 2025-06-29 04:21:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5b04558d]) (logid:c99dacfb) ClusterDRS.poll is being called at 2025-06-29T02:22:00+0000 2025-06-29 04:21:48,355 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5b04558d]) (logid:c99dacfb) Removed 0 old drs migration plans 2025-06-29 04:21:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b5cf383]) (logid:a3269311) No inactive management server node found 2025-06-29 04:21:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0b5cf383]) (logid:a3269311) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:50,316 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0970bc8d]) (logid:3de939e6) HA health check task is running... 2025-06-29 04:21:50,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8577: Seq 7-8577: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:21:50,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8577 2025-06-29 04:21:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e507b79]) (logid:a82ea4c3) No inactive management server node found 2025-06-29 04:21:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e507b79]) (logid:a82ea4c3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:52,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-528b3693]) (logid:40746db6) No inactive management server node found 2025-06-29 04:21:52,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-528b3693]) (logid:40746db6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f9b7f58]) (logid:c0898867) No inactive management server node found 2025-06-29 04:21:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f9b7f58]) (logid:c0898867) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:54,317 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3a62800b]) (logid:236118f4) HA health check task is running... 2025-06-29 04:21:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22272a2c]) (logid:d9de1fe3) No inactive management server node found 2025-06-29 04:21:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22272a2c]) (logid:d9de1fe3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-214810ee]) (logid:acc5fa65) No inactive management server node found 2025-06-29 04:21:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-214810ee]) (logid:acc5fa65) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5dc55741]) (logid:464f92fe) Enabled non-edge zones available for scan: 1 2025-06-29 04:21:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5dc55741]) (logid:464f92fe) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:21:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5dc55741]) (logid:464f92fe) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:21:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-67e1f0e2]) (logid:6c5eae50) Enabled non-edge zones available for scan: 1 2025-06-29 04:21:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-67e1f0e2]) (logid:6c5eae50) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:21:58,319 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-626dd83b]) (logid:e5d11406) HA health check task is running... 2025-06-29 04:21:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1c32b29]) (logid:2ddde535) No inactive management server node found 2025-06-29 04:21:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1c32b29]) (logid:2ddde535) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:21:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9adcf6]) (logid:b57e8fc2) No inactive management server node found 2025-06-29 04:21:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b9adcf6]) (logid:b57e8fc2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:22:00,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8578: Seq 7-8578: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:00,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8578 2025-06-29 04:22:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4f0242c]) (logid:6d66eeb2) No inactive management server node found 2025-06-29 04:22:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4f0242c]) (logid:6d66eeb2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:02,320 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c058ebb4]) (logid:577f0dcf) HA health check task is running... 2025-06-29 04:22:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-141696cb]) (logid:5ab30bd6) No inactive management server node found 2025-06-29 04:22:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-141696cb]) (logid:5ab30bd6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b255024]) (logid:278517e7) No inactive management server node found 2025-06-29 04:22:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b255024]) (logid:278517e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:05,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8579: Seq 7-8579: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:05,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8579 2025-06-29 04:22:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79cd92cb]) (logid:bc987303) No inactive management server node found 2025-06-29 04:22:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79cd92cb]) (logid:bc987303) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:06,322 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dfa455d7]) (logid:760d8a74) HA health check task is running... 2025-06-29 04:22:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0539e144]) (logid:e6c1ff66) No inactive management server node found 2025-06-29 04:22:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0539e144]) (logid:e6c1ff66) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d22fd501]) (logid:8019e7b7) No inactive management server node found 2025-06-29 04:22:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d22fd501]) (logid:8019e7b7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:10,323 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a6e37d9c]) (logid:d0e62d71) HA health check task is running... 2025-06-29 04:22:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ed121de]) (logid:a885d94c) No inactive management server node found 2025-06-29 04:22:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ed121de]) (logid:a885d94c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-088cd747]) (logid:beb3b40a) No inactive management server node found 2025-06-29 04:22:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-088cd747]) (logid:beb3b40a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:13,004 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-93fb8870]) (logid:ed7e076f) Checking if any host reservation can be released ... 2025-06-29 04:22:13,009 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-93fb8870]) (logid:ed7e076f) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-93fb8870]) (logid:ed7e076f) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-93fb8870]) (logid:ed7e076f) Done running HostReservationReleaseChecker ... 2025-06-29 04:22:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-64126f94]) (logid:70954869) Found 0 routers to update status. 2025-06-29 04:22:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-64126f94]) (logid:70954869) Found 0 VPC's to update Redundant State. 2025-06-29 04:22:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-64126f94]) (logid:70954869) Found 0 networks to update RvR status. 2025-06-29 04:22:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Running Capacity Checker ... 2025-06-29 04:22:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) recalculating system capacity 2025-06-29 04:22:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Executing cpu/ram capacity update 2025-06-29 04:22:13,169 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:13,177 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:13,182 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:22:13,183 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:22:13,183 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:22:13,195 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:13,196 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:13,202 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:22:13,204 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:22:13,204 DEBUG [c.c.c.CapacityManagerImpl] (pool-30-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:22:13,213 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Done executing cpu/ram capacity update 2025-06-29 04:22:13,213 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Executing storage capacity update 2025-06-29 04:22:13,221 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:22:13,222 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:22:13,222 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:22:13,224 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:22:13,230 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:22:13,230 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:22:13,230 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:22:13,233 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:22:13,239 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:22:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:22:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:22:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:22:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:22:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:22:13,251 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:22:13,257 DEBUG [c.c.s.StorageManagerImpl] (pool-31-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:22:13,257 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Done executing storage capacity update 2025-06-29 04:22:13,257 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Executing capacity updates for public ip and Vlans 2025-06-29 04:22:13,278 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Done capacity updates for public ip and Vlans 2025-06-29 04:22:13,278 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Executing capacity updates for private ip 2025-06-29 04:22:13,282 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Done executing capacity updates for private ip 2025-06-29 04:22:13,282 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Done recalculating system capacity 2025-06-29 04:22:13,318 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-b896c623]) (logid:f6c85845) Done running Capacity Checker ... 2025-06-29 04:22:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a7cf889]) (logid:e80352dd) No inactive management server node found 2025-06-29 04:22:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a7cf889]) (logid:e80352dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:14,325 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6ea04a40]) (logid:bb954c28) HA health check task is running... 2025-06-29 04:22:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2836fe74]) (logid:e4517bae) No inactive management server node found 2025-06-29 04:22:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2836fe74]) (logid:e4517bae) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:15,327 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8580: Seq 7-8580: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:15,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8580 2025-06-29 04:22:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a53abeaa]) (logid:ee12b2c5) No inactive management server node found 2025-06-29 04:22:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a53abeaa]) (logid:ee12b2c5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9d6c1d9]) (logid:827eb932) No inactive management server node found 2025-06-29 04:22:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9d6c1d9]) (logid:827eb932) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:18,022 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-f6fde24d]) (logid:381953e6) AutoScaling Monitor is running... 2025-06-29 04:22:18,088 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-f98b2145]) (logid:c4517884) Starting to remove all volume_stats rows older than [Sat Jun 28 16:22:18 CEST 2025]. 2025-06-29 04:22:18,089 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-f98b2145]) (logid:c4517884) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:22:18 CEST 2025]. 2025-06-29 04:22:18,327 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fa116976]) (logid:5fc70e2f) HA health check task is running... 2025-06-29 04:22:18,517 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-a1ea66ef]) (logid:75211db3) Starting to remove all vm_stats rows older than [Sat Jun 28 16:22:18 CEST 2025]. 2025-06-29 04:22:18,545 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-a1ea66ef]) (logid:75211db3) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:22:18 CEST 2025]. 2025-06-29 04:22:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d88bf5c]) (logid:e9aad3a9) No inactive management server node found 2025-06-29 04:22:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d88bf5c]) (logid:e9aad3a9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:19,552 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:22:19,553 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:22:19,554 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) Seq 2-4725683383995203779: Routed from 90520738888109 2025-06-29 04:22:19,598 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) Seq 1-4725683383995203779: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:22:19,599 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:22:19,600 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) Seq 2-3926857400090362067: Routed from 90520738888109 2025-06-29 04:22:19,644 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-7200ea93]) (logid:f3bebe4f) Seq 2-3926857400090362067: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:22:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f746438]) (logid:55ed1ea7) No inactive management server node found 2025-06-29 04:22:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f746438]) (logid:55ed1ea7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:22,210 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:22:22,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:22:22,214 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) Seq 2-4725683383995203780: Routed from 90520738888109 2025-06-29 04:22:22,268 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) Seq 1-4725683383995203780: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:22:22,307 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:22:22,308 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) Seq 2-3926857400090362068: Routed from 90520738888109 2025-06-29 04:22:22,328 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8706aa76]) (logid:e8e97527) HA health check task is running... 2025-06-29 04:22:22,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a62f0d5]) (logid:35c8915a) No inactive management server node found 2025-06-29 04:22:22,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a62f0d5]) (logid:35c8915a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:22,411 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-0c3f4ae5]) (logid:994cb8d4) Seq 2-3926857400090362068: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:22:22,952 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) StorageCollector is running... 2025-06-29 04:22:22,958 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:22:22,958 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:22:22,958 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:22:22,959 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 2-3938116399158788149: Routed from 90520738888109 2025-06-29 04:22:23,013 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 4-3938116399158788149: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:22:23,017 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:22:23,017 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:22:23,017 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:22:23,018 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 2-4725683383995203781: Routed from 90520738888109 2025-06-29 04:22:23,069 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 1-4725683383995203781: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:22:23,081 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:22:23,081 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:22:23,081 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:22:23,082 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 2-4725683383995203782: Routed from 90520738888109 2025-06-29 04:22:23,170 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 1-4725683383995203782: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:22:23,182 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:22:23,182 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:22:23,182 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:22:23,183 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 2-4725683383995203783: Routed from 90520738888109 2025-06-29 04:22:23,267 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 1-4725683383995203783: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:22:23,269 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:22:23,270 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:22:23,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:22:23,271 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 2-3926857400090362069: Routed from 90520738888109 2025-06-29 04:22:23,274 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b7ef70e]) (logid:e6ff38ce) Seq 2-3926857400090362069: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:22:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd6b0da1]) (logid:f3480cb5) No inactive management server node found 2025-06-29 04:22:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd6b0da1]) (logid:f3480cb5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:25,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8581: Seq 7-8581: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:25,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8581 2025-06-29 04:22:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32276357]) (logid:34215f21) No inactive management server node found 2025-06-29 04:22:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32276357]) (logid:34215f21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:26,329 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5f5cd718]) (logid:0667592d) HA health check task is running... 2025-06-29 04:22:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0de49648]) (logid:eb13a483) No inactive management server node found 2025-06-29 04:22:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0de49648]) (logid:eb13a483) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7a85cb60]) (logid:057947a8) Enabled non-edge zones available for scan: 1 2025-06-29 04:22:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-7a85cb60]) (logid:057947a8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:22:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-7a85cb60]) (logid:057947a8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:22:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-18903246]) (logid:b5ce51fd) Enabled non-edge zones available for scan: 1 2025-06-29 04:22:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-18903246]) (logid:b5ce51fd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:22:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d1d5ea8]) (logid:a67ed13d) No inactive management server node found 2025-06-29 04:22:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d1d5ea8]) (logid:a67ed13d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c19feeb]) (logid:02a6c342) No inactive management server node found 2025-06-29 04:22:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c19feeb]) (logid:02a6c342) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:30,331 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-593ea057]) (logid:42d0578a) HA health check task is running... 2025-06-29 04:22:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f03bd997]) (logid:c9c58904) No inactive management server node found 2025-06-29 04:22:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f03bd997]) (logid:c9c58904) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fcb7660]) (logid:8f02e836) No inactive management server node found 2025-06-29 04:22:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fcb7660]) (logid:8f02e836) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:34,332 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fb948263]) (logid:bc025f4e) HA health check task is running... 2025-06-29 04:22:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a3bad67]) (logid:672d2957) No inactive management server node found 2025-06-29 04:22:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a3bad67]) (logid:672d2957) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:35,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8582: Seq 7-8582: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:35,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8582 2025-06-29 04:22:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2aaa9f8]) (logid:712525f2) No inactive management server node found 2025-06-29 04:22:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2aaa9f8]) (logid:712525f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99b9f58d]) (logid:19e19ee9) No inactive management server node found 2025-06-29 04:22:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99b9f58d]) (logid:19e19ee9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:38,334 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-6c40f556]) (logid:7943576f) HA health check task is running... 2025-06-29 04:22:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ff13cf7]) (logid:735db61b) No inactive management server node found 2025-06-29 04:22:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ff13cf7]) (logid:735db61b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82d32814]) (logid:632e86b4) No inactive management server node found 2025-06-29 04:22:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82d32814]) (logid:632e86b4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13514ed9]) (logid:80e0f384) No inactive management server node found 2025-06-29 04:22:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13514ed9]) (logid:80e0f384) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:42,335 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b5bf8d5b]) (logid:5bd24d1c) HA health check task is running... 2025-06-29 04:22:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-de68c155]) (logid:73970ade) DbCollector is running... 2025-06-29 04:22:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2211fbeb]) (logid:8b62af57) Found 0 routers to update status. 2025-06-29 04:22:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2211fbeb]) (logid:8b62af57) Found 0 VPC's to update Redundant State. 2025-06-29 04:22:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2211fbeb]) (logid:8b62af57) Found 0 networks to update RvR status. 2025-06-29 04:22:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ef0bcf4]) (logid:765d2fe5) No inactive management server node found 2025-06-29 04:22:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ef0bcf4]) (logid:765d2fe5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) ManagementServerCollector is running... 2025-06-29 04:22:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) DB properties were already loaded 2025-06-29 04:22:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:22:44,059 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Execution of process [4066846] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:22:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Exit value of process [4066846] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:22:44,059 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Process [4066846] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:22:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) usage install: null 2025-06-29 04:22:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Sessions active: 0 2025-06-29 04:22:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:22:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066849] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:22:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:22:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066854] for command [/bin/bash -c uname -r ]. 2025-06-29 04:22:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:22:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066855] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:22:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) system memory from /proc: 4887515136 2025-06-29 04:22:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:22:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066860] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:22:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) free memory from /proc: 297070592 2025-06-29 04:22:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:22:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066865] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:22:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) used memory from /proc: 1071740 2025-06-29 04:22:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:22:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066866] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:22:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:22:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066867] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:22:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:22:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066868] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:22:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:22:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066875] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:22:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:22:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Successfully executed process [4066876] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail buffersdirect.count, 93 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail buffersdirect.used, 321663 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail buffersmapped.capacity, 0 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail buffersmapped.count, 0 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail buffersmapped.used, 0 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:22:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail jvmuptime, 2823836 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memoryheap.committed, 620232704 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memoryheap.init, 77594624 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memoryheap.usage, 0.22629712447135272 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorynon-heap.committed, 197328896 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorynon-heap.max, -1 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorynon-heap.usage, -1.9500996E8 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorynon-heap.used, 195009960 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1151245084644095 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2235999100209951 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.Metaspace.usage, 0.9930897863051471 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.PS-Eden-Space.usage, 0.38114595640634313 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorytotal.committed, 817561600 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorytotal.init, 85262336 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorytotal.max, 1908932607 2025-06-29 04:22:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail memorytotal.used, 626995920 2025-06-29 04:22:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail threadsdeadlocks, [] 2025-06-29 04:22:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail threadsnew.count, 0 2025-06-29 04:22:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:22:44,136 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-910f8d6f]) (logid:025f3243) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:22:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":189017408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2823762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":626021568,"totalCommitted":817364992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431985960,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297070592,"systemMemoryUsed":1071740,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.12,0.36],"systemCyclesUsage":[66413936,53834313,4589691433],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:22:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 93, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:22:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":189017408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2823762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":626021568,"totalCommitted":817364992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431985960,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297070592,"systemMemoryUsed":1071740,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.12,0.36],"systemCyclesUsage":[66413936,53834313,4589691433],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:22:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:22:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-22:[ctx-189289aa]) (logid:318431ca) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:22:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":189017408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2823762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":626021568,"totalCommitted":817364992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431985960,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297070592,"systemMemoryUsed":1071740,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.12,0.36],"systemCyclesUsage":[66413936,53834313,4589691433],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:22:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:22:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 93, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:22:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":189017408,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2823762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":626021568,"totalCommitted":817364992,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":431985960,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":297070592,"systemMemoryUsed":1071740,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.12,0.36],"systemCyclesUsage":[66413936,53834313,4589691433],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:22:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6479c8fc]) (logid:ff4c8442) No inactive management server node found 2025-06-29 04:22:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6479c8fc]) (logid:ff4c8442) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:45,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8583: Seq 7-8583: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:45,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8583 2025-06-29 04:22:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:22:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:22:46,336 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4ca94a6b]) (logid:b193036e) HA health check task is running... 2025-06-29 04:22:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8edcf12c]) (logid:0cfcd03d) No inactive management server node found 2025-06-29 04:22:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8edcf12c]) (logid:0cfcd03d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:46,643 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:22:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:22:47,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:47,094 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:22:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:22:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfa7a0be]) (logid:d40106fb) No inactive management server node found 2025-06-29 04:22:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfa7a0be]) (logid:d40106fb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-974f4671]) (logid:a2f2cc8d) VM scheduler.poll is being called at 2025-06-29T02:23:00+0000 2025-06-29 04:22:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-974f4671]) (logid:a2f2cc8d) Got 0 scheduled jobs to be executed at 2025-06-29T02:23:00+0000 2025-06-29 04:22:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-974f4671]) (logid:a2f2cc8d) Cleaned up 0 VM scheduled job entries 2025-06-29 04:22:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-0e9f0f3e]) (logid:beb4d5a1) VM Operation Thread Running 2025-06-29 04:22:48,349 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-19b4fb82]) (logid:5616b0a4) ClusterDRS.poll is being called at 2025-06-29T02:23:00+0000 2025-06-29 04:22:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-19b4fb82]) (logid:5616b0a4) Removed 0 old drs migration plans 2025-06-29 04:22:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6391233]) (logid:eb27f29f) No inactive management server node found 2025-06-29 04:22:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6391233]) (logid:eb27f29f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:50,338 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-647e2ee6]) (logid:4e5e4fb5) HA health check task is running... 2025-06-29 04:22:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-caa794e8]) (logid:2c433a67) No inactive management server node found 2025-06-29 04:22:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-caa794e8]) (logid:2c433a67) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf36753d]) (logid:ec5fa314) No inactive management server node found 2025-06-29 04:22:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf36753d]) (logid:ec5fa314) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f4b3841]) (logid:a92de762) No inactive management server node found 2025-06-29 04:22:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f4b3841]) (logid:a92de762) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:54,340 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-abd7a988]) (logid:16738dfa) HA health check task is running... 2025-06-29 04:22:55,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8585: Seq 7-8585: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:22:55,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8585 2025-06-29 04:22:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce8fe327]) (logid:260e00d1) No inactive management server node found 2025-06-29 04:22:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce8fe327]) (logid:260e00d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be5994a0]) (logid:be452bef) No inactive management server node found 2025-06-29 04:22:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be5994a0]) (logid:be452bef) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ee93f228]) (logid:ee6f16f2) Enabled non-edge zones available for scan: 1 2025-06-29 04:22:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ee93f228]) (logid:ee6f16f2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:22:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ee93f228]) (logid:ee6f16f2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:22:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-42daa7dd]) (logid:51ec4421) Enabled non-edge zones available for scan: 1 2025-06-29 04:22:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-42daa7dd]) (logid:51ec4421) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:22:58,341 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-359148f1]) (logid:70dfd5a7) HA health check task is running... 2025-06-29 04:22:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c4741a1]) (logid:da3ed22e) No inactive management server node found 2025-06-29 04:22:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c4741a1]) (logid:da3ed22e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:22:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85c34bf1]) (logid:4795a403) No inactive management server node found 2025-06-29 04:22:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85c34bf1]) (logid:4795a403) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-344d47a1]) (logid:0a4b8368) No inactive management server node found 2025-06-29 04:23:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-344d47a1]) (logid:0a4b8368) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:23:02,342 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d13e5050]) (logid:d2592dc3) HA health check task is running... 2025-06-29 04:23:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54d448c0]) (logid:0173c38d) No inactive management server node found 2025-06-29 04:23:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54d448c0]) (logid:0173c38d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b34dd75]) (logid:4080469a) No inactive management server node found 2025-06-29 04:23:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b34dd75]) (logid:4080469a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:05,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8586: Seq 7-8586: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:23:05,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8586 2025-06-29 04:23:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55713fe2]) (logid:2c875275) No inactive management server node found 2025-06-29 04:23:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55713fe2]) (logid:2c875275) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:06,343 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ec71143c]) (logid:94978fb0) HA health check task is running... 2025-06-29 04:23:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53198c44]) (logid:81629ea3) No inactive management server node found 2025-06-29 04:23:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53198c44]) (logid:81629ea3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1deea7f]) (logid:4d1be9c3) No inactive management server node found 2025-06-29 04:23:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1deea7f]) (logid:4d1be9c3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:10,344 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8c902853]) (logid:1cbb43f4) HA health check task is running... 2025-06-29 04:23:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea095b32]) (logid:350b9dd9) No inactive management server node found 2025-06-29 04:23:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea095b32]) (logid:350b9dd9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a09e68d]) (logid:82146c4d) No inactive management server node found 2025-06-29 04:23:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a09e68d]) (logid:82146c4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-05b5657e]) (logid:93f07ac6) Found 0 routers to update status. 2025-06-29 04:23:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-05b5657e]) (logid:93f07ac6) Found 0 VPC's to update Redundant State. 2025-06-29 04:23:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-05b5657e]) (logid:93f07ac6) Found 0 networks to update RvR status. 2025-06-29 04:23:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1236acf4]) (logid:f81335b8) No inactive management server node found 2025-06-29 04:23:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1236acf4]) (logid:f81335b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:14,346 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-12de039f]) (logid:0762222e) HA health check task is running... 2025-06-29 04:23:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f74efeb]) (logid:9c38e094) No inactive management server node found 2025-06-29 04:23:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f74efeb]) (logid:9c38e094) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:15,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8587: Seq 7-8587: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:23:15,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8587 2025-06-29 04:23:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-877cff9e]) (logid:9f8c7aed) No inactive management server node found 2025-06-29 04:23:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-877cff9e]) (logid:9f8c7aed) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-607cbc82]) (logid:eb1f6f20) No inactive management server node found 2025-06-29 04:23:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-607cbc82]) (logid:eb1f6f20) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:18,024 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-9e699010]) (logid:3eef6092) AutoScaling Monitor is running... 2025-06-29 04:23:18,091 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-4c1b2d3f]) (logid:cb0caf15) Starting to remove all volume_stats rows older than [Sat Jun 28 16:23:18 CEST 2025]. 2025-06-29 04:23:18,092 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-4c1b2d3f]) (logid:cb0caf15) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:23:18 CEST 2025]. 2025-06-29 04:23:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-89896ebc]) (logid:08abd976) Resetting hosts suitable for reconnect 2025-06-29 04:23:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-89896ebc]) (logid:08abd976) Completed resetting hosts suitable for reconnect 2025-06-29 04:23:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-89896ebc]) (logid:08abd976) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:23:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-89896ebc]) (logid:08abd976) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:23:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-89896ebc]) (logid:08abd976) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:23:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-89896ebc]) (logid:08abd976) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:23:18,347 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5b4175c6]) (logid:ccd3a2e6) HA health check task is running... 2025-06-29 04:23:18,546 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-c4232a6e]) (logid:b5d4e13f) Starting to remove all vm_stats rows older than [Sat Jun 28 16:23:18 CEST 2025]. 2025-06-29 04:23:18,571 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-c4232a6e]) (logid:b5d4e13f) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:23:18 CEST 2025]. 2025-06-29 04:23:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd212828]) (logid:c127823d) No inactive management server node found 2025-06-29 04:23:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd212828]) (logid:c127823d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:19,647 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:23:19,648 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:23:19,651 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) Seq 2-4725683383995203784: Routed from 90520738888109 2025-06-29 04:23:19,695 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) Seq 1-4725683383995203784: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:23:19,696 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:23:19,697 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) Seq 2-3926857400090362070: Routed from 90520738888109 2025-06-29 04:23:19,741 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-64ad2e1d]) (logid:b7b2be96) Seq 2-3926857400090362070: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:23:20,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae2c5a9f]) (logid:864c6fac) No inactive management server node found 2025-06-29 04:23:20,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae2c5a9f]) (logid:864c6fac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:22,348 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ebe693f1]) (logid:0ffb57ed) HA health check task is running... 2025-06-29 04:23:22,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96811622]) (logid:a63f0151) No inactive management server node found 2025-06-29 04:23:22,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96811622]) (logid:a63f0151) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:22,475 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:23:22,479 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:23:22,480 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) Seq 2-4725683383995203785: Routed from 90520738888109 2025-06-29 04:23:22,535 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) Seq 1-4725683383995203785: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:23:22,570 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:23:22,571 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) Seq 2-3926857400090362071: Routed from 90520738888109 2025-06-29 04:23:22,667 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-64032164]) (logid:3d713f3d) Seq 2-3926857400090362071: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:23:23,275 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) StorageCollector is running... 2025-06-29 04:23:23,281 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:23:23,281 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:23:23,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:23:23,282 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3938116399158788150: Routed from 90520738888109 2025-06-29 04:23:23,336 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 4-3938116399158788150: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:23:23,339 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:23:23,339 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:23:23,339 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:23:23,341 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3926857400090362072: Routed from 90520738888109 2025-06-29 04:23:23,355 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3926857400090362072: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:23:23,367 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:23:23,367 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:23:23,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:23:23,368 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3926857400090362073: Routed from 90520738888109 2025-06-29 04:23:23,460 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3926857400090362073: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:23:23,470 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:23:23,470 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:23:23,470 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:23:23,471 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-4725683383995203786: Routed from 90520738888109 2025-06-29 04:23:23,515 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 1-4725683383995203786: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:23:23,517 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:23:23,517 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:23:23,517 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:23:23,519 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3926857400090362074: Routed from 90520738888109 2025-06-29 04:23:23,562 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-35bd61ea]) (logid:05268b7d) Seq 2-3926857400090362074: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:23:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62208c54]) (logid:0e6a74ef) No inactive management server node found 2025-06-29 04:23:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62208c54]) (logid:0e6a74ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:25,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8588: Seq 7-8588: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:23:25,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8588 2025-06-29 04:23:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25f67c31]) (logid:1bae17e4) No inactive management server node found 2025-06-29 04:23:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25f67c31]) (logid:1bae17e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:26,349 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3c433379]) (logid:9e5908b4) HA health check task is running... 2025-06-29 04:23:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-582695cf]) (logid:da07f885) No inactive management server node found 2025-06-29 04:23:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-582695cf]) (logid:da07f885) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bb973cd2]) (logid:40d0df6d) Enabled non-edge zones available for scan: 1 2025-06-29 04:23:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bb973cd2]) (logid:40d0df6d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:23:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bb973cd2]) (logid:40d0df6d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:23:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5dc67e05]) (logid:2600b62d) Enabled non-edge zones available for scan: 1 2025-06-29 04:23:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5dc67e05]) (logid:2600b62d) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:23:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7e6c670]) (logid:6b5a1b31) No inactive management server node found 2025-06-29 04:23:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7e6c670]) (logid:6b5a1b31) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9e02b57]) (logid:ee29f6a7) No inactive management server node found 2025-06-29 04:23:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9e02b57]) (logid:ee29f6a7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:30,351 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ccef30c8]) (logid:475bc016) HA health check task is running... 2025-06-29 04:23:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a686cfb]) (logid:6a77fe35) No inactive management server node found 2025-06-29 04:23:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a686cfb]) (logid:6a77fe35) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ff7a173]) (logid:830d59c5) No inactive management server node found 2025-06-29 04:23:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ff7a173]) (logid:830d59c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:34,352 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-74442f2c]) (logid:8384ac37) HA health check task is running... 2025-06-29 04:23:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cec57a8]) (logid:679d6cdc) No inactive management server node found 2025-06-29 04:23:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cec57a8]) (logid:679d6cdc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:35,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8589: Seq 7-8589: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:23:35,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8589 2025-06-29 04:23:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5102f714]) (logid:6aad0d75) No inactive management server node found 2025-06-29 04:23:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5102f714]) (logid:6aad0d75) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4225e79a]) (logid:57b1e3f7) No inactive management server node found 2025-06-29 04:23:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4225e79a]) (logid:57b1e3f7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:38,353 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4b539600]) (logid:9474d94e) HA health check task is running... 2025-06-29 04:23:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2c21950]) (logid:9e4be919) No inactive management server node found 2025-06-29 04:23:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2c21950]) (logid:9e4be919) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:40,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8590: Seq 7-8590: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:23:40,341 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8590 2025-06-29 04:23:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9776c7f0]) (logid:5b512fbb) No inactive management server node found 2025-06-29 04:23:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9776c7f0]) (logid:5b512fbb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb4aaefc]) (logid:97f690e7) No inactive management server node found 2025-06-29 04:23:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb4aaefc]) (logid:97f690e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:42,354 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7896ecdd]) (logid:e5a297ad) HA health check task is running... 2025-06-29 04:23:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-73f2f07e]) (logid:62d36af3) DbCollector is running... 2025-06-29 04:23:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-462b6399]) (logid:4066f19d) Found 0 routers to update status. 2025-06-29 04:23:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-462b6399]) (logid:4066f19d) Found 0 VPC's to update Redundant State. 2025-06-29 04:23:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-462b6399]) (logid:4066f19d) Found 0 networks to update RvR status. 2025-06-29 04:23:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e29e20c8]) (logid:47b44d6a) No inactive management server node found 2025-06-29 04:23:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e29e20c8]) (logid:47b44d6a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) ManagementServerCollector is running... 2025-06-29 04:23:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) DB properties were already loaded 2025-06-29 04:23:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:23:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Execution of process [4066883] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:23:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Exit value of process [4066883] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:23:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Process [4066883] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:23:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) usage install: null 2025-06-29 04:23:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Sessions active: 0 2025-06-29 04:23:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:23:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066886] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:23:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:23:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066891] for command [/bin/bash -c uname -r ]. 2025-06-29 04:23:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:23:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066892] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:23:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) system memory from /proc: 4887515136 2025-06-29 04:23:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:23:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066897] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:23:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) free memory from /proc: 294957056 2025-06-29 04:23:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:23:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066902] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:23:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) used memory from /proc: 1071836 2025-06-29 04:23:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:23:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066903] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:23:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:23:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066904] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:23:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:23:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066905] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:23:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:23:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066912] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:23:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:23:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Successfully executed process [4066913] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:23:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail buffersdirect.count, 93 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail buffersdirect.used, 321663 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail buffersmapped.capacity, 0 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail buffersmapped.count, 0 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail buffersmapped.used, 0 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail jvmuptime, 2883831 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memoryheap.committed, 620232704 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memoryheap.init, 77594624 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memoryheap.usage, 0.23811088882609732 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorynon-heap.committed, 197394432 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorynon-heap.max, -1 2025-06-29 04:23:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorynon-heap.usage, -1.95073136E8 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorynon-heap.used, 195073136 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11515158457744601 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22401127236978038 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.Metaspace.usage, 0.9931564797794118 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.PS-Eden-Space.usage, 0.4153382584855363 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorytotal.committed, 817627136 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorytotal.init, 85262336 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorytotal.max, 1908932607 2025-06-29 04:23:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail memorytotal.used, 649610776 2025-06-29 04:23:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail threadsdeadlocks, [] 2025-06-29 04:23:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail threadsnew.count, 0 2025-06-29 04:23:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:23:44,132 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-88da4a81]) (logid:a1e5e966) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:23:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":166605080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2883761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":648700760,"totalCommitted":817627136,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":454537640,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":294957056,"systemMemoryUsed":1071836,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.11,0.34],"systemCyclesUsage":[66414264,53834612,4589714547],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:23:44,133 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 95, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:23:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":166605080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2883761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":648700760,"totalCommitted":817627136,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":454537640,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":294957056,"systemMemoryUsed":1071836,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.11,0.34],"systemCyclesUsage":[66414264,53834612,4589714547],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:23:44,133 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:23:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-22:[ctx-a5dc5448]) (logid:f35c3510) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:23:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":166605080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2883761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":648700760,"totalCommitted":817627136,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":454537640,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":294957056,"systemMemoryUsed":1071836,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.11,0.34],"systemCyclesUsage":[66414264,53834612,4589714547],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:23:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:23:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 95, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:23:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":166605080,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2883761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":648700760,"totalCommitted":817627136,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":454537640,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":294957056,"systemMemoryUsed":1071836,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.11,0.34],"systemCyclesUsage":[66414264,53834612,4589714547],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:23:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f806e4f2]) (logid:3064934b) No inactive management server node found 2025-06-29 04:23:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f806e4f2]) (logid:3064934b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:23:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:23:46,356 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2ae65172]) (logid:5ee96700) HA health check task is running... 2025-06-29 04:23:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-000ae109]) (logid:64013dad) No inactive management server node found 2025-06-29 04:23:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-000ae109]) (logid:64013dad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:23:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:23:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:23:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:23:47,071 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:23:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:23:47,075 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:23:47,078 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:23:47,891 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86087a28]) (logid:ce22dc6a) No inactive management server node found 2025-06-29 04:23:47,891 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86087a28]) (logid:ce22dc6a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-976bc19b]) (logid:13448ca7) VM scheduler.poll is being called at 2025-06-29T02:24:00+0000 2025-06-29 04:23:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-976bc19b]) (logid:13448ca7) Got 0 scheduled jobs to be executed at 2025-06-29T02:24:00+0000 2025-06-29 04:23:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-976bc19b]) (logid:13448ca7) Cleaned up 0 VM scheduled job entries 2025-06-29 04:23:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-6d65d406]) (logid:5028a72b) VM Operation Thread Running 2025-06-29 04:23:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-873d1d8b]) (logid:7f67f580) ClusterDRS.poll is being called at 2025-06-29T02:24:00+0000 2025-06-29 04:23:48,355 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-873d1d8b]) (logid:7f67f580) Removed 0 old drs migration plans 2025-06-29 04:23:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca818195]) (logid:20887460) No inactive management server node found 2025-06-29 04:23:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca818195]) (logid:20887460) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:50,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8592: Seq 7-8592: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:23:50,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8592 2025-06-29 04:23:50,356 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d3cac91a]) (logid:8f9efacb) HA health check task is running... 2025-06-29 04:23:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc969b93]) (logid:d081ce4d) No inactive management server node found 2025-06-29 04:23:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc969b93]) (logid:d081ce4d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e9f510d]) (logid:7fc1e690) No inactive management server node found 2025-06-29 04:23:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e9f510d]) (logid:7fc1e690) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-030ac3dc]) (logid:00cec8bd) No inactive management server node found 2025-06-29 04:23:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-030ac3dc]) (logid:00cec8bd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:54,358 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-afaa9e55]) (logid:5cdf89e6) HA health check task is running... 2025-06-29 04:23:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff21fd48]) (logid:dbdcabf1) No inactive management server node found 2025-06-29 04:23:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ff21fd48]) (logid:dbdcabf1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36720300]) (logid:a85c4a53) No inactive management server node found 2025-06-29 04:23:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36720300]) (logid:a85c4a53) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-03bb9b33]) (logid:cf6bcb34) Enabled non-edge zones available for scan: 1 2025-06-29 04:23:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-03bb9b33]) (logid:cf6bcb34) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:23:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-03bb9b33]) (logid:cf6bcb34) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:23:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9df042fc]) (logid:b71f2793) Enabled non-edge zones available for scan: 1 2025-06-29 04:23:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9df042fc]) (logid:b71f2793) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:23:58,359 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3cdea0d8]) (logid:afb7bda7) HA health check task is running... 2025-06-29 04:23:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-feb544ae]) (logid:d1ed7529) No inactive management server node found 2025-06-29 04:23:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-feb544ae]) (logid:d1ed7529) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:23:59,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e49a3e2a]) (logid:8b1ea07e) No inactive management server node found 2025-06-29 04:23:59,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e49a3e2a]) (logid:8b1ea07e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:00,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8593: Seq 7-8593: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:00,342 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8593 2025-06-29 04:24:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29335392]) (logid:64d2d9c7) No inactive management server node found 2025-06-29 04:24:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29335392]) (logid:64d2d9c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:02,360 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1072c5d7]) (logid:01867f2b) HA health check task is running... 2025-06-29 04:24:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11a32dc3]) (logid:11d17a29) No inactive management server node found 2025-06-29 04:24:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11a32dc3]) (logid:11d17a29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:24:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3361ffb]) (logid:767343e3) No inactive management server node found 2025-06-29 04:24:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3361ffb]) (logid:767343e3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7867f34]) (logid:3c3618a3) No inactive management server node found 2025-06-29 04:24:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7867f34]) (logid:3c3618a3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:06,361 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e866523d]) (logid:25e0b913) HA health check task is running... 2025-06-29 04:24:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51b46fbc]) (logid:7e411577) No inactive management server node found 2025-06-29 04:24:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51b46fbc]) (logid:7e411577) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-716d57c0]) (logid:cecfc783) No inactive management server node found 2025-06-29 04:24:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-716d57c0]) (logid:cecfc783) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:10,333 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8594: Seq 7-8594: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:10,343 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8594 2025-06-29 04:24:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8b161fe]) (logid:491c3a84) No inactive management server node found 2025-06-29 04:24:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8b161fe]) (logid:491c3a84) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:10,363 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-fba06f96]) (logid:b2bc7eba) HA health check task is running... 2025-06-29 04:24:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b53b065]) (logid:1f2ef50d) No inactive management server node found 2025-06-29 04:24:11,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b53b065]) (logid:1f2ef50d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a3fe6912]) (logid:ff795d82) Found 0 routers to update status. 2025-06-29 04:24:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a3fe6912]) (logid:ff795d82) Found 0 VPC's to update Redundant State. 2025-06-29 04:24:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a3fe6912]) (logid:ff795d82) Found 0 networks to update RvR status. 2025-06-29 04:24:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0f5bce4]) (logid:269892e6) No inactive management server node found 2025-06-29 04:24:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0f5bce4]) (logid:269892e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:14,364 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a5e168ea]) (logid:58adf938) HA health check task is running... 2025-06-29 04:24:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f181ca8a]) (logid:e22e1a12) No inactive management server node found 2025-06-29 04:24:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f181ca8a]) (logid:e22e1a12) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4481047]) (logid:8a1c65b8) No inactive management server node found 2025-06-29 04:24:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4481047]) (logid:8a1c65b8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b6c8838]) (logid:4948f9ff) No inactive management server node found 2025-06-29 04:24:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b6c8838]) (logid:4948f9ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:18,027 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-15447045]) (logid:2c6fd8e3) AutoScaling Monitor is running... 2025-06-29 04:24:18,094 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-162305fb]) (logid:46541d45) Starting to remove all volume_stats rows older than [Sat Jun 28 16:24:18 CEST 2025]. 2025-06-29 04:24:18,094 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-162305fb]) (logid:46541d45) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:24:18 CEST 2025]. 2025-06-29 04:24:18,365 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-781c8944]) (logid:39711a11) HA health check task is running... 2025-06-29 04:24:18,572 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-961992b2]) (logid:99064c3c) Starting to remove all vm_stats rows older than [Sat Jun 28 16:24:18 CEST 2025]. 2025-06-29 04:24:18,597 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-961992b2]) (logid:99064c3c) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:24:18 CEST 2025]. 2025-06-29 04:24:19,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-851d1fdd]) (logid:ee503638) No inactive management server node found 2025-06-29 04:24:19,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-851d1fdd]) (logid:ee503638) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:19,744 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:24:19,746 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:24:19,747 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) Seq 2-4725683383995203787: Routed from 90520738888109 2025-06-29 04:24:19,791 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) Seq 1-4725683383995203787: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:24:19,792 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:24:19,793 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) Seq 2-3926857400090362075: Routed from 90520738888109 2025-06-29 04:24:19,837 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-13439016]) (logid:8aceb45e) Seq 2-3926857400090362075: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:24:20,332 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8595: Seq 7-8595: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:20,341 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8595 2025-06-29 04:24:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df447ed9]) (logid:0a255735) No inactive management server node found 2025-06-29 04:24:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df447ed9]) (logid:0a255735) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e16c4166]) (logid:a9ee9ad1) No inactive management server node found 2025-06-29 04:24:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e16c4166]) (logid:a9ee9ad1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:22,366 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4d15cd0d]) (logid:a2b5f7e8) HA health check task is running... 2025-06-29 04:24:22,721 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:24:22,723 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:24:22,724 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) Seq 2-4725683383995203788: Routed from 90520738888109 2025-06-29 04:24:22,778 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) Seq 1-4725683383995203788: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:24:22,818 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:24:22,819 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) Seq 2-3926857400090362076: Routed from 90520738888109 2025-06-29 04:24:22,907 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-609a9c45]) (logid:b6f73b8e) Seq 2-3926857400090362076: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:24:23,563 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) StorageCollector is running... 2025-06-29 04:24:23,569 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:24:23,569 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:24:23,569 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:24:23,570 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-3938116399158788151: Routed from 90520738888109 2025-06-29 04:24:23,625 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 4-3938116399158788151: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:24:23,628 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:24:23,628 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:24:23,628 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:24:23,629 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-4725683383995203789: Routed from 90520738888109 2025-06-29 04:24:23,678 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 1-4725683383995203789: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:24:23,689 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:24:23,689 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:24:23,689 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:24:23,690 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-3926857400090362077: Routed from 90520738888109 2025-06-29 04:24:23,701 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-3926857400090362077: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:24:23,712 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:24:23,712 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:24:23,712 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:24:23,713 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-4725683383995203790: Routed from 90520738888109 2025-06-29 04:24:23,798 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 1-4725683383995203790: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:24:23,799 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:24:23,800 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:24:23,800 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:24:23,801 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-3926857400090362078: Routed from 90520738888109 2025-06-29 04:24:23,845 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-04b145d2]) (logid:b8010376) Seq 2-3926857400090362078: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:24:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69fc0c82]) (logid:d7d18eda) No inactive management server node found 2025-06-29 04:24:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69fc0c82]) (logid:d7d18eda) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b3fb51a]) (logid:d23354b3) No inactive management server node found 2025-06-29 04:24:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b3fb51a]) (logid:d23354b3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:26,367 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a60fd9b9]) (logid:15d0bcb1) HA health check task is running... 2025-06-29 04:24:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4b97b78]) (logid:8c8ace4a) No inactive management server node found 2025-06-29 04:24:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4b97b78]) (logid:8c8ace4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-335627b3]) (logid:1fc08907) Enabled non-edge zones available for scan: 1 2025-06-29 04:24:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-335627b3]) (logid:1fc08907) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:24:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-335627b3]) (logid:1fc08907) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:24:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e1e32522]) (logid:67e11207) Enabled non-edge zones available for scan: 1 2025-06-29 04:24:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e1e32522]) (logid:67e11207) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:24:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bad443d5]) (logid:14def921) No inactive management server node found 2025-06-29 04:24:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bad443d5]) (logid:14def921) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8458655c]) (logid:3e944974) No inactive management server node found 2025-06-29 04:24:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8458655c]) (logid:3e944974) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:30,334 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8596: Seq 7-8596: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:30,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8596 2025-06-29 04:24:30,369 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-87c515ff]) (logid:0e3cff65) HA health check task is running... 2025-06-29 04:24:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e788c807]) (logid:ecf84de2) No inactive management server node found 2025-06-29 04:24:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e788c807]) (logid:ecf84de2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1436f750]) (logid:7e2b17c6) No inactive management server node found 2025-06-29 04:24:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1436f750]) (logid:7e2b17c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:34,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e7fc965]) (logid:00cd8a5b) No inactive management server node found 2025-06-29 04:24:34,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e7fc965]) (logid:00cd8a5b) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:34,370 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-919b0c6a]) (logid:da0a0fcb) HA health check task is running... 2025-06-29 04:24:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccfab6fc]) (logid:5d602241) No inactive management server node found 2025-06-29 04:24:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ccfab6fc]) (logid:5d602241) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:37,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99f77f23]) (logid:02d19223) No inactive management server node found 2025-06-29 04:24:37,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99f77f23]) (logid:02d19223) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:38,371 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e56d362c]) (logid:b5d11fac) HA health check task is running... 2025-06-29 04:24:38,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bae7699]) (logid:94ee576d) No inactive management server node found 2025-06-29 04:24:38,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bae7699]) (logid:94ee576d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:40,334 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8597: Seq 7-8597: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:40,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8597 2025-06-29 04:24:40,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-538cd953]) (logid:23c89c68) No inactive management server node found 2025-06-29 04:24:40,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-538cd953]) (logid:23c89c68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d5a22a9]) (logid:8f94a160) No inactive management server node found 2025-06-29 04:24:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d5a22a9]) (logid:8f94a160) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:42,372 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-93c64bb4]) (logid:0fdfdb34) HA health check task is running... 2025-06-29 04:24:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-1d7d4aac]) (logid:a34fc6cd) DbCollector is running... 2025-06-29 04:24:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4a0cd48d]) (logid:bb7a2332) Found 0 routers to update status. 2025-06-29 04:24:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4a0cd48d]) (logid:bb7a2332) Found 0 VPC's to update Redundant State. 2025-06-29 04:24:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4a0cd48d]) (logid:bb7a2332) Found 0 networks to update RvR status. 2025-06-29 04:24:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce24ffab]) (logid:727dc7f1) No inactive management server node found 2025-06-29 04:24:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce24ffab]) (logid:727dc7f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) ManagementServerCollector is running... 2025-06-29 04:24:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) DB properties were already loaded 2025-06-29 04:24:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:24:44,057 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Execution of process [4066927] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:24:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Exit value of process [4066927] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:24:44,057 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Process [4066927] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:24:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) usage install: null 2025-06-29 04:24:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Sessions active: 0 2025-06-29 04:24:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:24:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066930] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:24:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:24:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066935] for command [/bin/bash -c uname -r ]. 2025-06-29 04:24:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:24:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066936] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:24:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) system memory from /proc: 4887515136 2025-06-29 04:24:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:24:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066941] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:24:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) free memory from /proc: 300322816 2025-06-29 04:24:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:24:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066946] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:24:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) used memory from /proc: 1072092 2025-06-29 04:24:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:24:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066947] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:24:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:24:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066948] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:24:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:24:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066949] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:24:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:24:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066956] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:24:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:24:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Successfully executed process [4066957] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:24:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:24:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail buffersdirect.count, 93 2025-06-29 04:24:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail buffersdirect.used, 321663 2025-06-29 04:24:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail buffersmapped.capacity, 0 2025-06-29 04:24:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail buffersmapped.count, 0 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail buffersmapped.used, 0 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail gcPS-Scavenge.count, 107 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail gcPS-Scavenge.time, 553 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail jvmuptime, 2943828 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memoryheap.committed, 620232704 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memoryheap.init, 77594624 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memoryheap.usage, 0.24935745033907453 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorynon-heap.committed, 197459968 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorynon-heap.max, -1 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorynon-heap.usage, -1.95167312E8 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorynon-heap.used, 195167312 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11542026292988536 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22444138034458627 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.Metaspace.usage, 0.9932159926470588 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.PS-Eden-Space.usage, 0.44816052022912733 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.PS-Old-Gen.usage, 0.1261318993193027 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorypools.PS-Survivor-Space.usage, 0.0 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorytotal.committed, 817692672 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorytotal.init, 85262336 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorytotal.max, 1908932607 2025-06-29 04:24:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail memorytotal.used, 671353016 2025-06-29 04:24:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail threadsdeadlocks, [] 2025-06-29 04:24:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail threadsnew.count, 0 2025-06-29 04:24:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:24:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-930ab4f9]) (logid:32aa4afb) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:24:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":144821008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2943760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":670579008,"totalCommitted":817692672,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":476006568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":300322816,"systemMemoryUsed":1072092,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.32],"systemCyclesUsage":[66414597,53834907,4589737665],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:24:44,129 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 97, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:24:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":144821008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2943760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":670579008,"totalCommitted":817692672,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":476006568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":300322816,"systemMemoryUsed":1072092,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.32],"systemCyclesUsage":[66414597,53834907,4589737665],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:24:44,129 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:24:44,134 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-22:[ctx-2c52ce55]) (logid:15a64017) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:24:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":144821008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2943760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":670579008,"totalCommitted":817692672,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":476006568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":300322816,"systemMemoryUsed":1072092,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.32],"systemCyclesUsage":[66414597,53834907,4589737665],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:24:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:24:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 97, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:24:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":620232704,"freeJvmMemoryBytes":144821008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":2943760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":670579008,"totalCommitted":817692672,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":476006568,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":300322816,"systemMemoryUsed":1072092,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.09,0.32],"systemCyclesUsage":[66414597,53834907,4589737665],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:24:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b76a093]) (logid:6b7c23b7) No inactive management server node found 2025-06-29 04:24:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b76a093]) (logid:6b7c23b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:45,334 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8598: Seq 7-8598: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:45,343 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8598 2025-06-29 04:24:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:24:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:24:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8a8cf2d]) (logid:e8547aa7) No inactive management server node found 2025-06-29 04:24:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8a8cf2d]) (logid:e8547aa7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:46,374 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2028d663]) (logid:d4c264d6) HA health check task is running... 2025-06-29 04:24:46,648 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:24:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:24:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:24:46,653 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:24:47,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:24:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:24:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:24:47,097 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:24:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb450984]) (logid:05d64999) No inactive management server node found 2025-06-29 04:24:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb450984]) (logid:05d64999) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-77733cb5]) (logid:d4509ce7) Resetting hosts suitable for reconnect 2025-06-29 04:24:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-77733cb5]) (logid:d4509ce7) Completed resetting hosts suitable for reconnect 2025-06-29 04:24:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-77733cb5]) (logid:d4509ce7) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:24:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-77733cb5]) (logid:d4509ce7) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:24:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-77733cb5]) (logid:d4509ce7) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:24:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-77733cb5]) (logid:d4509ce7) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:24:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2ca5ff15]) (logid:f18a420b) VM scheduler.poll is being called at 2025-06-29T02:25:00+0000 2025-06-29 04:24:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2ca5ff15]) (logid:f18a420b) Got 0 scheduled jobs to be executed at 2025-06-29T02:25:00+0000 2025-06-29 04:24:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2ca5ff15]) (logid:f18a420b) Cleaned up 0 VM scheduled job entries 2025-06-29 04:24:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-fe51306d]) (logid:6aeb7da2) VM Operation Thread Running 2025-06-29 04:24:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6655a7ee]) (logid:6afefdc5) ClusterDRS.poll is being called at 2025-06-29T02:25:00+0000 2025-06-29 04:24:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6655a7ee]) (logid:6afefdc5) Removed 0 old drs migration plans 2025-06-29 04:24:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68f08612]) (logid:f5f7217e) No inactive management server node found 2025-06-29 04:24:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68f08612]) (logid:f5f7217e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:50,375 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-13fb32c9]) (logid:5e036181) HA health check task is running... 2025-06-29 04:24:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-077d898e]) (logid:bd11f76f) No inactive management server node found 2025-06-29 04:24:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-077d898e]) (logid:bd11f76f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18a1d677]) (logid:05587833) No inactive management server node found 2025-06-29 04:24:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18a1d677]) (logid:05587833) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d57a552]) (logid:69df0f9f) No inactive management server node found 2025-06-29 04:24:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d57a552]) (logid:69df0f9f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:54,377 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-45e70c2f]) (logid:da6a05ae) HA health check task is running... 2025-06-29 04:24:55,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8600: Seq 7-8600: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:24:55,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8600 2025-06-29 04:24:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92e708ea]) (logid:1d5501d6) No inactive management server node found 2025-06-29 04:24:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92e708ea]) (logid:1d5501d6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-067e0577]) (logid:6b21b919) No inactive management server node found 2025-06-29 04:24:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-067e0577]) (logid:6b21b919) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8081fae5]) (logid:9af33cb9) Enabled non-edge zones available for scan: 1 2025-06-29 04:24:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8081fae5]) (logid:9af33cb9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:24:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8081fae5]) (logid:9af33cb9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:24:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-71af642d]) (logid:ed8e6111) Enabled non-edge zones available for scan: 1 2025-06-29 04:24:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-71af642d]) (logid:ed8e6111) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:24:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5df338e]) (logid:b7abb4cc) No inactive management server node found 2025-06-29 04:24:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5df338e]) (logid:b7abb4cc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:24:58,378 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-8f1a6e4a]) (logid:b3171c66) HA health check task is running... 2025-06-29 04:24:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2100b944]) (logid:cfe4c185) No inactive management server node found 2025-06-29 04:24:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2100b944]) (logid:cfe4c185) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aaa5a02e]) (logid:94f4407b) No inactive management server node found 2025-06-29 04:25:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aaa5a02e]) (logid:94f4407b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:02,379 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-52246a69]) (logid:506b0c21) HA health check task is running... 2025-06-29 04:25:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d19d0bb]) (logid:bd0c5ed8) No inactive management server node found 2025-06-29 04:25:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d19d0bb]) (logid:bd0c5ed8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-598b6856]) (logid:225db593) No inactive management server node found 2025-06-29 04:25:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-598b6856]) (logid:225db593) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:05,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8601: Seq 7-8601: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:25:05,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8601 2025-06-29 04:25:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d408e10e]) (logid:e69d1d44) No inactive management server node found 2025-06-29 04:25:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d408e10e]) (logid:e69d1d44) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:06,380 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a5de7eab]) (logid:27e83caa) HA health check task is running... 2025-06-29 04:25:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bffe8a81]) (logid:2e81243b) No inactive management server node found 2025-06-29 04:25:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bffe8a81]) (logid:2e81243b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1089ec8]) (logid:3e989ba6) No inactive management server node found 2025-06-29 04:25:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1089ec8]) (logid:3e989ba6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f122c56]) (logid:65bfeff9) No inactive management server node found 2025-06-29 04:25:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f122c56]) (logid:65bfeff9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:10,382 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-98c60b90]) (logid:c47c5db2) HA health check task is running... 2025-06-29 04:25:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b49c6db7]) (logid:9cca332e) No inactive management server node found 2025-06-29 04:25:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b49c6db7]) (logid:9cca332e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b6dfca88]) (logid:411e71aa) Found 0 routers to update status. 2025-06-29 04:25:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b6dfca88]) (logid:411e71aa) Found 0 VPC's to update Redundant State. 2025-06-29 04:25:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b6dfca88]) (logid:411e71aa) Found 0 networks to update RvR status. 2025-06-29 04:25:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daeb3d8e]) (logid:4658d977) No inactive management server node found 2025-06-29 04:25:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-daeb3d8e]) (logid:4658d977) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:14,384 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b1129bd0]) (logid:dc19f215) HA health check task is running... 2025-06-29 04:25:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12196460]) (logid:3e42cfd3) No inactive management server node found 2025-06-29 04:25:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12196460]) (logid:3e42cfd3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:15,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8602: Seq 7-8602: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:25:15,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8602 2025-06-29 04:25:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4248d5f4]) (logid:7f2365fe) No inactive management server node found 2025-06-29 04:25:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4248d5f4]) (logid:7f2365fe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad15d23]) (logid:6a3b6e72) No inactive management server node found 2025-06-29 04:25:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad15d23]) (logid:6a3b6e72) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:18,029 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-f3933c7a]) (logid:d889d62e) AutoScaling Monitor is running... 2025-06-29 04:25:18,096 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-c18903e4]) (logid:f37c0926) Starting to remove all volume_stats rows older than [Sat Jun 28 16:25:18 CEST 2025]. 2025-06-29 04:25:18,097 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-c18903e4]) (logid:f37c0926) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:25:18 CEST 2025]. 2025-06-29 04:25:18,385 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-72f23bf0]) (logid:d40fa35e) HA health check task is running... 2025-06-29 04:25:18,599 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-36262f4a]) (logid:700f8c19) Starting to remove all vm_stats rows older than [Sat Jun 28 16:25:18 CEST 2025]. 2025-06-29 04:25:18,621 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-36262f4a]) (logid:700f8c19) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:25:18 CEST 2025]. 2025-06-29 04:25:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-464a37f6]) (logid:592d6dbf) No inactive management server node found 2025-06-29 04:25:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-464a37f6]) (logid:592d6dbf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:19,840 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:25:19,842 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:25:19,844 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) Seq 2-4725683383995203791: Routed from 90520738888109 2025-06-29 04:25:19,888 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) Seq 1-4725683383995203791: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:25:19,889 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:25:19,890 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) Seq 2-3926857400090362079: Routed from 90520738888109 2025-06-29 04:25:19,934 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c5767aa8]) (logid:cbb9d028) Seq 2-3926857400090362079: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:25:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9015437]) (logid:22a6d48c) No inactive management server node found 2025-06-29 04:25:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9015437]) (logid:22a6d48c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58fbe5fc]) (logid:808f063a) No inactive management server node found 2025-06-29 04:25:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58fbe5fc]) (logid:808f063a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:22,386 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-285f2359]) (logid:53d77ebd) HA health check task is running... 2025-06-29 04:25:22,970 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:25:22,973 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:25:22,974 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) Seq 2-4725683383995203792: Routed from 90520738888109 2025-06-29 04:25:23,050 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) Seq 1-4725683383995203792: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:25:23,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:25:23,088 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) Seq 2-3926857400090362080: Routed from 90520738888109 2025-06-29 04:25:23,194 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4954ce72]) (logid:ae6b57ab) Seq 2-3926857400090362080: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:25:23,846 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) StorageCollector is running... 2025-06-29 04:25:23,851 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:25:23,851 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:25:23,852 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:25:23,852 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-3938116399158788152: Routed from 90520738888109 2025-06-29 04:25:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b64fbcc]) (logid:05379b69) No inactive management server node found 2025-06-29 04:25:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b64fbcc]) (logid:05379b69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:23,907 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 4-3938116399158788152: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:25:23,910 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:25:23,910 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:25:23,910 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:25:23,911 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-4725683383995203793: Routed from 90520738888109 2025-06-29 04:25:23,959 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 1-4725683383995203793: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:25:23,970 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:25:23,970 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:25:23,970 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:25:23,971 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-3926857400090362081: Routed from 90520738888109 2025-06-29 04:25:23,983 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-3926857400090362081: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:25:23,994 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:25:23,994 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:25:23,994 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:25:23,995 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-4725683383995203794: Routed from 90520738888109 2025-06-29 04:25:24,078 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 1-4725683383995203794: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:25:24,080 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:25:24,080 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:25:24,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:25:24,081 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-3926857400090362082: Routed from 90520738888109 2025-06-29 04:25:24,130 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-0b6ecc6b]) (logid:670a32b9) Seq 2-3926857400090362082: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:25:25,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8603: Seq 7-8603: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:25:25,344 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8603 2025-06-29 04:25:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-243ecee4]) (logid:39fc184a) No inactive management server node found 2025-06-29 04:25:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-243ecee4]) (logid:39fc184a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:26,388 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cd0995e5]) (logid:f54ef086) HA health check task is running... 2025-06-29 04:25:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-522bd9c8]) (logid:9463492c) No inactive management server node found 2025-06-29 04:25:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-522bd9c8]) (logid:9463492c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2d5db1ad]) (logid:23521875) Enabled non-edge zones available for scan: 1 2025-06-29 04:25:28,099 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:25:28,101 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:25:28,103 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-2d5db1ad]) (logid:23521875) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:25:28,105 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2d5db1ad]) (logid:23521875) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:25:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a0539bc7]) (logid:23790174) Enabled non-edge zones available for scan: 1 2025-06-29 04:25:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a0539bc7]) (logid:23790174) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:25:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cb56c97]) (logid:938a782c) No inactive management server node found 2025-06-29 04:25:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cb56c97]) (logid:938a782c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa0a376f]) (logid:a9a2e082) No inactive management server node found 2025-06-29 04:25:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa0a376f]) (logid:a9a2e082) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:30,389 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-066c910d]) (logid:2d5751b1) HA health check task is running... 2025-06-29 04:25:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed8c4b05]) (logid:fb9dcd78) No inactive management server node found 2025-06-29 04:25:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed8c4b05]) (logid:fb9dcd78) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39f660bd]) (logid:f929f861) No inactive management server node found 2025-06-29 04:25:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39f660bd]) (logid:f929f861) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19086bdd]) (logid:5bb1905d) No inactive management server node found 2025-06-29 04:25:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19086bdd]) (logid:5bb1905d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:34,390 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-472d1000]) (logid:f549f804) HA health check task is running... 2025-06-29 04:25:35,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8604: Seq 7-8604: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:25:35,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8604 2025-06-29 04:25:35,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3711e4ce]) (logid:ed78d6d5) No inactive management server node found 2025-06-29 04:25:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3711e4ce]) (logid:ed78d6d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:25:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc991e2]) (logid:ed8febf5) No inactive management server node found 2025-06-29 04:25:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc991e2]) (logid:ed8febf5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:38,392 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1b68585a]) (logid:73c33355) HA health check task is running... 2025-06-29 04:25:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-866c4570]) (logid:e68bdc74) No inactive management server node found 2025-06-29 04:25:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-866c4570]) (logid:e68bdc74) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f39ca0de]) (logid:96aba3c7) No inactive management server node found 2025-06-29 04:25:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f39ca0de]) (logid:96aba3c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a41ad124]) (logid:ba5f04d8) No inactive management server node found 2025-06-29 04:25:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a41ad124]) (logid:ba5f04d8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:42,393 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4f5ad91b]) (logid:243ce946) HA health check task is running... 2025-06-29 04:25:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-7d84054b]) (logid:d669fdcb) DbCollector is running... 2025-06-29 04:25:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1408b5a9]) (logid:de44539c) Found 0 routers to update status. 2025-06-29 04:25:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1408b5a9]) (logid:de44539c) Found 0 VPC's to update Redundant State. 2025-06-29 04:25:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1408b5a9]) (logid:de44539c) Found 0 networks to update RvR status. 2025-06-29 04:25:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40031e12]) (logid:4f1ea96a) No inactive management server node found 2025-06-29 04:25:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40031e12]) (logid:4f1ea96a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) ManagementServerCollector is running... 2025-06-29 04:25:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) DB properties were already loaded 2025-06-29 04:25:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:25:44,059 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Execution of process [4066962] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:25:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Exit value of process [4066962] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:25:44,060 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Process [4066962] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:25:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) usage install: null 2025-06-29 04:25:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Sessions active: 0 2025-06-29 04:25:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:25:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066965] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:25:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:25:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066970] for command [/bin/bash -c uname -r ]. 2025-06-29 04:25:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:25:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066971] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:25:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) system memory from /proc: 4887515136 2025-06-29 04:25:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:25:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066976] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:25:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) free memory from /proc: 344858624 2025-06-29 04:25:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:25:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066981] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:25:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) used memory from /proc: 1024040 2025-06-29 04:25:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:25:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066982] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:25:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:25:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066983] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:25:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:25:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066984] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:25:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:25:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066991] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:25:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:25:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Successfully executed process [4066992] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:25:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail buffersdirect.count, 93 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail buffersdirect.used, 321663 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail buffersmapped.capacity, 0 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail buffersmapped.count, 0 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail buffersmapped.used, 0 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail jvmuptime, 3003830 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memoryheap.committed, 638058496 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memoryheap.init, 77594624 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memoryheap.usage, 0.10242524391934951 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorynon-heap.committed, 197591040 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorynon-heap.max, -1 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorynon-heap.usage, -1.95302136E8 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorynon-heap.used, 195302136 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11595657824580112 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22495792648382043 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.Metaspace.usage, 0.9932548828125 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.PS-Eden-Space.usage, 0.013844791653635408 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:25:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorytotal.committed, 835649536 2025-06-29 04:25:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorytotal.init, 85262336 2025-06-29 04:25:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorytotal.max, 1908932607 2025-06-29 04:25:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail memorytotal.used, 390825024 2025-06-29 04:25:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail threadsdeadlocks, [] 2025-06-29 04:25:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail threadsnew.count, 0 2025-06-29 04:25:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:25:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-e7af3e35]) (logid:311126e0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:25:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":443262792,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3003763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":390080944,"totalCommitted":835649536,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195522888,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344858624,"systemMemoryUsed":1024040,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.3],"systemCyclesUsage":[66414937,53835203,4589760777],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:25:44,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 99, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:25:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":443262792,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3003763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":390080944,"totalCommitted":835649536,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195522888,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344858624,"systemMemoryUsed":1024040,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.3],"systemCyclesUsage":[66414937,53835203,4589760777],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:25:44,131 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:25:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-23:[ctx-6ca4bb5c]) (logid:91bd8ab5) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:25:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":443262792,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3003763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":390080944,"totalCommitted":835649536,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195522888,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344858624,"systemMemoryUsed":1024040,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.3],"systemCyclesUsage":[66414937,53835203,4589760777],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:25:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:25:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 99, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:25:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":443262792,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3003763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":390080944,"totalCommitted":835649536,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":195522888,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344858624,"systemMemoryUsed":1024040,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.08,0.3],"systemCyclesUsage":[66414937,53835203,4589760777],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:25:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd782e9a]) (logid:c5b0341c) No inactive management server node found 2025-06-29 04:25:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd782e9a]) (logid:c5b0341c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:45,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8605: Seq 7-8605: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:25:45,351 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8605 2025-06-29 04:25:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:25:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:25:46,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad2313c9]) (logid:984e5508) No inactive management server node found 2025-06-29 04:25:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad2313c9]) (logid:984e5508) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:46,395 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4a2e44bc]) (logid:5f5cf1a1) HA health check task is running... 2025-06-29 04:25:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:25:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:25:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:25:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:25:47,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:25:47,097 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:25:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:25:47,100 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:25:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88342dcc]) (logid:91cbac54) No inactive management server node found 2025-06-29 04:25:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88342dcc]) (logid:91cbac54) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-732e1c4e]) (logid:3cbd39d9) VM scheduler.poll is being called at 2025-06-29T02:26:00+0000 2025-06-29 04:25:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-732e1c4e]) (logid:3cbd39d9) Got 0 scheduled jobs to be executed at 2025-06-29T02:26:00+0000 2025-06-29 04:25:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-732e1c4e]) (logid:3cbd39d9) Cleaned up 0 VM scheduled job entries 2025-06-29 04:25:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-3ddc4027]) (logid:fc9b8600) VM Operation Thread Running 2025-06-29 04:25:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-983a416c]) (logid:95468860) ClusterDRS.poll is being called at 2025-06-29T02:26:00+0000 2025-06-29 04:25:48,355 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-983a416c]) (logid:95468860) Removed 0 old drs migration plans 2025-06-29 04:25:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-568d1bbc]) (logid:30ce5267) No inactive management server node found 2025-06-29 04:25:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-568d1bbc]) (logid:30ce5267) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:50,336 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8607: Seq 7-8607: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:25:50,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8607 2025-06-29 04:25:50,396 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ee8d1126]) (logid:dec8bfa3) HA health check task is running... 2025-06-29 04:25:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a77345a]) (logid:77aea383) No inactive management server node found 2025-06-29 04:25:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a77345a]) (logid:77aea383) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:52,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4173c2c]) (logid:dadcfec4) No inactive management server node found 2025-06-29 04:25:52,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4173c2c]) (logid:dadcfec4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4cd8979]) (logid:51883ffd) No inactive management server node found 2025-06-29 04:25:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4cd8979]) (logid:51883ffd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:54,397 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-dfc29496]) (logid:08c19633) HA health check task is running... 2025-06-29 04:25:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7eba3f0]) (logid:38e53f78) No inactive management server node found 2025-06-29 04:25:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7eba3f0]) (logid:38e53f78) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a6e71d2]) (logid:e0b2a219) No inactive management server node found 2025-06-29 04:25:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a6e71d2]) (logid:e0b2a219) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b06acc6f]) (logid:9acb776d) Enabled non-edge zones available for scan: 1 2025-06-29 04:25:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b06acc6f]) (logid:9acb776d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:25:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b06acc6f]) (logid:9acb776d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:25:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-13cf9995]) (logid:723c092f) Enabled non-edge zones available for scan: 1 2025-06-29 04:25:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-13cf9995]) (logid:723c092f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:25:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05cf1f62]) (logid:71bfc85d) No inactive management server node found 2025-06-29 04:25:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05cf1f62]) (logid:71bfc85d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:25:58,398 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b33b618c]) (logid:c9b1b16b) HA health check task is running... 2025-06-29 04:25:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-905c5bd3]) (logid:5d318b51) No inactive management server node found 2025-06-29 04:25:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-905c5bd3]) (logid:5d318b51) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:00,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8608: Seq 7-8608: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:00,348 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8608 2025-06-29 04:26:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4adc98a9]) (logid:952a38fa) No inactive management server node found 2025-06-29 04:26:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4adc98a9]) (logid:952a38fa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:02,399 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-13981181]) (logid:09033243) HA health check task is running... 2025-06-29 04:26:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aed889cd]) (logid:c85b3262) No inactive management server node found 2025-06-29 04:26:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aed889cd]) (logid:c85b3262) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a2dcd51]) (logid:f9aca702) No inactive management server node found 2025-06-29 04:26:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a2dcd51]) (logid:f9aca702) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1fa9e87]) (logid:acc9da8c) No inactive management server node found 2025-06-29 04:26:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1fa9e87]) (logid:acc9da8c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:06,401 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d6638c5b]) (logid:78b1eafa) HA health check task is running... 2025-06-29 04:26:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e75b3378]) (logid:fd959ba4) No inactive management server node found 2025-06-29 04:26:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e75b3378]) (logid:fd959ba4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31104b07]) (logid:e4a25357) No inactive management server node found 2025-06-29 04:26:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31104b07]) (logid:e4a25357) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:10,337 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8609: Seq 7-8609: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:10,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8609 2025-06-29 04:26:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa14544f]) (logid:47d085d1) No inactive management server node found 2025-06-29 04:26:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa14544f]) (logid:47d085d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:10,402 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-185e8463]) (logid:9c121c76) HA health check task is running... 2025-06-29 04:26:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f0d1b31]) (logid:483e13a6) No inactive management server node found 2025-06-29 04:26:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f0d1b31]) (logid:483e13a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a919391b]) (logid:72dfde15) Found 0 routers to update status. 2025-06-29 04:26:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a919391b]) (logid:72dfde15) Found 0 VPC's to update Redundant State. 2025-06-29 04:26:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a919391b]) (logid:72dfde15) Found 0 networks to update RvR status. 2025-06-29 04:26:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44cc974c]) (logid:b29955a9) No inactive management server node found 2025-06-29 04:26:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44cc974c]) (logid:b29955a9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:14,403 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-71c07f82]) (logid:31a0dc66) HA health check task is running... 2025-06-29 04:26:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf65dfa2]) (logid:2b22ecb8) No inactive management server node found 2025-06-29 04:26:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf65dfa2]) (logid:2b22ecb8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32c7d6fa]) (logid:ff6c427d) No inactive management server node found 2025-06-29 04:26:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32c7d6fa]) (logid:ff6c427d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cab96731]) (logid:8de0ecff) No inactive management server node found 2025-06-29 04:26:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cab96731]) (logid:8de0ecff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:18,031 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-6b729d08]) (logid:966b6b2b) AutoScaling Monitor is running... 2025-06-29 04:26:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c7045bca]) (logid:2c92b491) Resetting hosts suitable for reconnect 2025-06-29 04:26:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c7045bca]) (logid:2c92b491) Completed resetting hosts suitable for reconnect 2025-06-29 04:26:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c7045bca]) (logid:2c92b491) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:26:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c7045bca]) (logid:2c92b491) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:26:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c7045bca]) (logid:2c92b491) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:26:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c7045bca]) (logid:2c92b491) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:26:18,098 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-07fa14eb]) (logid:fb2628e8) Starting to remove all volume_stats rows older than [Sat Jun 28 16:26:18 CEST 2025]. 2025-06-29 04:26:18,099 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-07fa14eb]) (logid:fb2628e8) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:26:18 CEST 2025]. 2025-06-29 04:26:18,405 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b7416703]) (logid:e063b6f5) HA health check task is running... 2025-06-29 04:26:18,623 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-4ccb3f85]) (logid:1f4ef2b5) Starting to remove all vm_stats rows older than [Sat Jun 28 16:26:18 CEST 2025]. 2025-06-29 04:26:18,645 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-4ccb3f85]) (logid:1f4ef2b5) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:26:18 CEST 2025]. 2025-06-29 04:26:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5240f4bb]) (logid:3a717a3e) No inactive management server node found 2025-06-29 04:26:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5240f4bb]) (logid:3a717a3e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:19,937 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:26:19,938 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:26:19,940 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) Seq 2-4725683383995203795: Routed from 90520738888109 2025-06-29 04:26:19,984 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) Seq 1-4725683383995203795: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:26:19,985 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:26:19,986 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) Seq 2-3926857400090362083: Routed from 90520738888109 2025-06-29 04:26:20,030 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-68282d86]) (logid:c4588484) Seq 2-3926857400090362083: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:26:20,352 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8610: Seq 7-8610: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:20,362 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8610 2025-06-29 04:26:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c58f77ae]) (logid:01d39b07) No inactive management server node found 2025-06-29 04:26:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c58f77ae]) (logid:01d39b07) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efce90db]) (logid:b22d5b94) No inactive management server node found 2025-06-29 04:26:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efce90db]) (logid:b22d5b94) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:22,408 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-38ff4ee3]) (logid:ddd391eb) HA health check task is running... 2025-06-29 04:26:23,269 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:26:23,272 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:26:23,274 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) Seq 2-4725683383995203796: Routed from 90520738888109 2025-06-29 04:26:23,328 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) Seq 1-4725683383995203796: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:26:23,363 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:26:23,364 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) Seq 2-3926857400090362084: Routed from 90520738888109 2025-06-29 04:26:23,456 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-5f142d6a]) (logid:d599c9ae) Seq 2-3926857400090362084: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:26:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17909129]) (logid:e29d10e8) No inactive management server node found 2025-06-29 04:26:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17909129]) (logid:e29d10e8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:24,130 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) StorageCollector is running... 2025-06-29 04:26:24,135 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:26:24,135 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:26:24,135 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:26:24,136 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3938116399158788153: Routed from 90520738888109 2025-06-29 04:26:24,193 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 4-3938116399158788153: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:26:24,198 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:26:24,198 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:26:24,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:26:24,199 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3926857400090362085: Routed from 90520738888109 2025-06-29 04:26:24,212 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3926857400090362085: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:26:24,222 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:26:24,222 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:26:24,222 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:26:24,223 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3926857400090362086: Routed from 90520738888109 2025-06-29 04:26:24,315 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3926857400090362086: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:26:24,328 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:26:24,328 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:26:24,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:26:24,330 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-4725683383995203797: Routed from 90520738888109 2025-06-29 04:26:24,373 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 1-4725683383995203797: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:26:24,375 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:26:24,375 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:26:24,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:26:24,376 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3926857400090362087: Routed from 90520738888109 2025-06-29 04:26:24,421 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-4a7b0d44]) (logid:8d8a3352) Seq 2-3926857400090362087: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:26:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6964b79]) (logid:c1c30877) No inactive management server node found 2025-06-29 04:26:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6964b79]) (logid:c1c30877) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:26,410 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-8d959bad]) (logid:e0188373) HA health check task is running... 2025-06-29 04:26:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63b1c517]) (logid:ea02c818) No inactive management server node found 2025-06-29 04:26:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63b1c517]) (logid:ea02c818) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-53fdaeba]) (logid:263e3f12) Enabled non-edge zones available for scan: 1 2025-06-29 04:26:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-53fdaeba]) (logid:263e3f12) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:26:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-53fdaeba]) (logid:263e3f12) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:26:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ff68967c]) (logid:0e154f5c) Enabled non-edge zones available for scan: 1 2025-06-29 04:26:28,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ff68967c]) (logid:0e154f5c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:26:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7323ef4]) (logid:c5ea0c1b) No inactive management server node found 2025-06-29 04:26:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e7323ef4]) (logid:c5ea0c1b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-101488a5]) (logid:335f2322) No inactive management server node found 2025-06-29 04:26:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-101488a5]) (logid:335f2322) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:30,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8611: Seq 7-8611: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:30,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8611 2025-06-29 04:26:30,411 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dde229b5]) (logid:7d1eb0c5) HA health check task is running... 2025-06-29 04:26:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c1b4a9b]) (logid:0f1d8edb) No inactive management server node found 2025-06-29 04:26:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c1b4a9b]) (logid:0f1d8edb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84079f6a]) (logid:ccd04107) No inactive management server node found 2025-06-29 04:26:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84079f6a]) (logid:ccd04107) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-145262a4]) (logid:4361fb7a) No inactive management server node found 2025-06-29 04:26:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-145262a4]) (logid:4361fb7a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:34,413 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c53aed8e]) (logid:60ced28a) HA health check task is running... 2025-06-29 04:26:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bd28c2b]) (logid:8c0d4d6b) No inactive management server node found 2025-06-29 04:26:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bd28c2b]) (logid:8c0d4d6b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad7d8d2]) (logid:d30ed0e3) No inactive management server node found 2025-06-29 04:26:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ad7d8d2]) (logid:d30ed0e3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:38,414 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-35cb71b2]) (logid:a0425988) HA health check task is running... 2025-06-29 04:26:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c8b72aa]) (logid:54b2cea8) No inactive management server node found 2025-06-29 04:26:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c8b72aa]) (logid:54b2cea8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:40,338 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8612: Seq 7-8612: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:40,349 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8612 2025-06-29 04:26:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-057ef4c2]) (logid:e57afe2e) No inactive management server node found 2025-06-29 04:26:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-057ef4c2]) (logid:e57afe2e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d68cf21b]) (logid:7cb4086a) No inactive management server node found 2025-06-29 04:26:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d68cf21b]) (logid:7cb4086a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:42,415 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ec1cb54b]) (logid:c3a026ef) HA health check task is running... 2025-06-29 04:26:43,018 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-825b47ee]) (logid:033e9655) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 04:26:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-dd490d02]) (logid:82b3f9cc) DbCollector is running... 2025-06-29 04:26:43,094 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-3e5c27f2]) (logid:85fa212b) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:26:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-3e5c27f2]) (logid:85fa212b) Snapshot scheduler is being called at [2025-06-29T02:26:43+0000]. 2025-06-29 04:26:43,095 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-3e5c27f2]) (logid:85fa212b) There are [0] scheduled snapshots to be executed at [2025-06-29T02:26:43+0000]. 2025-06-29 04:26:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-a0540d93]) (logid:580c373e) Found 0 running routers. 2025-06-29 04:26:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85058c6b]) (logid:93a46ac7) Found 0 routers to update status. 2025-06-29 04:26:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85058c6b]) (logid:93a46ac7) Found 0 VPC's to update Redundant State. 2025-06-29 04:26:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85058c6b]) (logid:93a46ac7) Found 0 networks to update RvR status. 2025-06-29 04:26:43,109 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Found 1 running routers. 2025-06-29 04:26:43,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:26:43,113 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Seq 2-3926857400090362088: Routed from 90520738888109 2025-06-29 04:26:43,113 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Seq 2-3926857400090362088: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:43,118 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-28957fa0]) (logid:4ca55c02) External devices are not used. Skipping external device usage collection 2025-06-29 04:26:43,133 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-c289a82d]) (logid:fabb1ca3) Started resource counters recalculation periodic task. 2025-06-29 04:26:43,136 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-c289a82d]) (logid:fabb1ca3) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:26:43,305 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-cb77544a]) (logid:8680707f) Backup backup.poll is being called at 2025-06-29T02:26:43+0000 2025-06-29 04:26:43,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd46411c]) (logid:35dd104d) No inactive management server node found 2025-06-29 04:26:43,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd46411c]) (logid:35dd104d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:43,538 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-c289a82d]) (logid:fabb1ca3) Finished resource counters recalculation periodic task. 2025-06-29 04:26:43,971 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-3926857400090362088: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:43,971 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Seq 2-3926857400090362088: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:26:43,971 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 04:26:43,974 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 04:26:44,020 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 04:26:44,021 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Seq 2-3926857400090362089: Routed from 90520738888109 2025-06-29 04:26:44,022 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Seq 2-3926857400090362089: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) ManagementServerCollector is running... 2025-06-29 04:26:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) DB properties were already loaded 2025-06-29 04:26:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:26:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Execution of process [4067007] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:26:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Exit value of process [4067007] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:26:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Process [4067007] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:26:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) usage install: null 2025-06-29 04:26:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Sessions active: 0 2025-06-29 04:26:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:26:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067010] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:26:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:26:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067015] for command [/bin/bash -c uname -r ]. 2025-06-29 04:26:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:26:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067016] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:26:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) system memory from /proc: 4887515136 2025-06-29 04:26:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:26:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067021] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:26:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) free memory from /proc: 334929920 2025-06-29 04:26:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:26:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067026] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:26:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) used memory from /proc: 1024104 2025-06-29 04:26:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:26:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067027] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:26:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:26:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067028] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:26:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:26:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067029] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:26:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:26:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067036] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:26:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:26:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Successfully executed process [4067037] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:26:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail buffersdirect.capacity, 321663 2025-06-29 04:26:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail buffersdirect.count, 93 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail buffersdirect.used, 321663 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail buffersmapped.capacity, 0 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail buffersmapped.count, 0 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail buffersmapped.used, 0 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail jvmuptime, 3063830 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memoryheap.committed, 638058496 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memoryheap.init, 77594624 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memoryheap.usage, 0.12351301193761158 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorynon-heap.committed, 197722112 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorynon-heap.max, -1 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorynon-heap.usage, -1.95402584E8 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorynon-heap.used, 195402584 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11613153159157558 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22556611790582198 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.Metaspace.usage, 0.9932849839154412 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.PS-Eden-Space.usage, 0.07531842124089896 2025-06-29 04:26:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:26:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:26:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorytotal.committed, 835780608 2025-06-29 04:26:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorytotal.init, 85262336 2025-06-29 04:26:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorytotal.max, 1908932607 2025-06-29 04:26:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail memorytotal.used, 431180600 2025-06-29 04:26:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail threadsdeadlocks, [] 2025-06-29 04:26:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail threadsnew.count, 0 2025-06-29 04:26:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:26:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-a7924530]) (logid:1bedc004) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:26:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":403032144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3063759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":430428936,"totalCommitted":835780608,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":235778016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":334929920,"systemMemoryUsed":1024104,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.08,0.28],"systemCyclesUsage":[66415290,53835523,4589783839],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:26:44,133 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 101, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:26:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":403032144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3063759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":430428936,"totalCommitted":835780608,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":235778016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":334929920,"systemMemoryUsed":1024104,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.08,0.28],"systemCyclesUsage":[66415290,53835523,4589783839],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:26:44,133 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:26:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-23:[ctx-8eff36c7]) (logid:c362ff87) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:26:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":403032144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3063759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":430428936,"totalCommitted":835780608,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":235778016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":334929920,"systemMemoryUsed":1024104,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.08,0.28],"systemCyclesUsage":[66415290,53835523,4589783839],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:26:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:26:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 101, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:26:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":403032144,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3063759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.06,"totalInit":85262336,"totalUsed":430428936,"totalCommitted":835780608,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":235778016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":334929920,"systemMemoryUsed":1024104,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.06,0.08,0.28],"systemCyclesUsage":[66415290,53835523,4589783839],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:26:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f8fc709]) (logid:21362196) No inactive management server node found 2025-06-29 04:26:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f8fc709]) (logid:21362196) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:26:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:26:45,942 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 2-3926857400090362089: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.53e1d2e1-1c4f-40e8-a91f-0e7391b5d3da","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:45,942 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-2430e085]) (logid:a05470b6) Seq 2-3926857400090362089: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 04:26:45,944 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 04:26:45,947 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:26:45,948 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Seq 2-3926857400090362090: Routed from 90520738888109 2025-06-29 04:26:45,948 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Seq 2-3926857400090362090: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b86f7095]) (logid:909338a9) No inactive management server node found 2025-06-29 04:26:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b86f7095]) (logid:909338a9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:46,413 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-3926857400090362090: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "9.603261947631836", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "20.316600799560547", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "33.38122367858887", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "42.490243911743164", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163841930", "lastRunDuration": "43.161869049072266", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163841999", "lastRunDuration": "8093.544960021973", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163850143", "lastRunDuration": "72.99375534057617", "message": "Memory Usage within limits with current at 50.2404%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163850216", "lastRunDuration": "3229.8595905303955", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163853493", "lastRunDuration": "41.016578674316406", "message": "Sufficient free space is 2571.30078125 MB"}, "lastRun": {"start": "2025-06-29 02:24:01.783902", "end": "2025-06-29 02:24:13.535038", "duration": "11.751136541366577"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163601323", "lastRunDuration": "56.92458152770996", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163601381", "lastRunDuration": "53.09629440307617", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163601565", "lastRunDuration": "45.62234878540039", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163601722", "lastRunDuration": "47.61958122253418", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:20:01.320750", "end": "2025-06-29 02:20:01.808874", "duration": "0.48812389373779297"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "9.603261947631836", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "20.316600799560547", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "33.38122367858887", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "42.490243911743164", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163841930", "lastRunDuration": "43.161869049072266", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163841999", "lastRunDuration": "8093.544960021973", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163850143", "lastRunDuration": "72.99375534057617", "message": "Memory Usage within limits with current at 50.2404%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163850216", "lastRunDuration": "3229.8595905303955", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163853493", "lastRunDuration": "41.016578674316406", "message": "Sufficient free space is 2571.30078125 MB"}, "lastRun": {"start": "2025-06-29 02:24:01.783902", "end": "2025-06-29 02:24:13.535038", "duration": "11.751136541366577"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163601323", "lastRunDuration": "56.92458152770996", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163601381", "lastRunDuration": "53.09629440307617", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163601565", "lastRunDuration": "45.62234878540039", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163601722", "lastRunDuration": "47.61958122253418", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:20:01.320750", "end": "2025-06-29 02:20:01.808874", "duration": "0.48812389373779297"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:46,413 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Seq 2-3926857400090362090: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:26:46,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "9.603261947631836", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "20.316600799560547", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "33.38122367858887", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "42.490243911743164", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163841930", "lastRunDuration": "43.161869049072266", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163841999", "lastRunDuration": "8093.544960021973", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163850143", "lastRunDuration": "72.99375534057617", "message": "Memory Usage within limits with current at 50.2404%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163850216", "lastRunDuration": "3229.8595905303955", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163853493", "lastRunDuration": "41.016578674316406", "message": "Sufficient free space is 2571.30078125 MB"}, "lastRun": {"start": "2025-06-29 02:24:01.783902", "end": "2025-06-29 02:24:13.535038", "duration": "11.751136541366577"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163601323", "lastRunDuration": "56.92458152770996", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163601381", "lastRunDuration": "53.09629440307617", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163601565", "lastRunDuration": "45.62234878540039", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163601722", "lastRunDuration": "47.61958122253418", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:20:01.320750", "end": "2025-06-29 02:20:01.808874", "duration": "0.48812389373779297"}}} 2025-06-29 04:26:46,416 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1ff769a7]) (logid:b52ef538) HA health check task is running... 2025-06-29 04:26:46,444 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "9.603261947631836", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "20.316600799560547", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "33.38122367858887", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751163841786", "lastRunDuration": "42.490243911743164", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751163841930", "lastRunDuration": "43.161869049072266", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751163841999", "lastRunDuration": "8093.544960021973", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751163850143", "lastRunDuration": "72.99375534057617", "message": "Memory Usage within limits with current at 50.2404%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751163850216", "lastRunDuration": "3229.8595905303955", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751163853493", "lastRunDuration": "41.016578674316406", "message": "Sufficient free space is 2571.30078125 MB"}, "lastRun": {"start": "2025-06-29 02:24:01.783902", "end": "2025-06-29 02:24:13.535038", "duration": "11.751136541366577"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751163601323", "lastRunDuration": "56.92458152770996", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751163601381", "lastRunDuration": "53.09629440307617", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751163601565", "lastRunDuration": "45.62234878540039", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751163601722", "lastRunDuration": "47.61958122253418", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:20:01.320750", "end": "2025-06-29 02:20:01.808874", "duration": "0.48812389373779297"}}} 2025-06-29 04:26:46,445 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found check types executed on VR basic, start: 2025-06-29 02:24:01.783902, end: 2025-06-29 02:24:13.535038, duration: 11.751136541366577 2025-06-29 04:26:46,455 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2390e436- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 04:24:01 CEST 2025, details: service is running which took running duration (ms) 9.603261947631836 2025-06-29 04:26:46,465 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7a12bb29- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 04:24:01 CEST 2025, details: service is running which took running duration (ms) 20.316600799560547 2025-06-29 04:26:46,476 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7fc51be7- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 04:24:01 CEST 2025, details: service is running which took running duration (ms) 33.38122367858887 2025-06-29 04:26:46,486 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7250aa32- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 04:24:01 CEST 2025, details: service is running which took running duration (ms) 42.490243911743164 2025-06-29 04:26:46,499 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@56c5511a- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 04:24:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 43.161869049072266 2025-06-29 04:26:46,510 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@39617318- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 04:24:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8093.544960021973 2025-06-29 04:26:46,520 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6eac5df0- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 04:24:10 CEST 2025, details: Memory Usage within limits with current at 50.2404% which took running duration (ms) 72.99375534057617 2025-06-29 04:26:46,532 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@19da6b2d- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 04:24:10 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3229.8595905303955 2025-06-29 04:26:46,542 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@656ba48d- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 04:24:13 CEST 2025, details: Sufficient free space is 2571.30078125 MB which took running duration (ms) 41.016578674316406 2025-06-29 04:26:46,542 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found check types executed on VR advanced, start: 2025-06-29 02:20:01.320750, end: 2025-06-29 02:20:01.808874, duration: 0.48812389373779297 2025-06-29 04:26:46,554 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5720f28b- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 04:20:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 56.92458152770996 2025-06-29 04:26:46,564 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5b6f6642- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 04:20:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 53.09629440307617 2025-06-29 04:26:46,574 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1126ed01- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 04:20:01 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 45.62234878540039 2025-06-29 04:26:46,585 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-88eb5bfa]) (logid:1df19063) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@29d17c64- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 04:20:01 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 47.61958122253418 2025-06-29 04:26:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:26:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:26:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:26:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:26:47,094 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:26:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:26:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:26:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:26:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28d7e8f0]) (logid:763fba73) No inactive management server node found 2025-06-29 04:26:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-28d7e8f0]) (logid:763fba73) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:48,052 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-1:[ctx-289ae0c7]) (logid:86fb38b1) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:26:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0573cae2]) (logid:d55a54a9) VM scheduler.poll is being called at 2025-06-29T02:27:00+0000 2025-06-29 04:26:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0573cae2]) (logid:d55a54a9) Got 0 scheduled jobs to be executed at 2025-06-29T02:27:00+0000 2025-06-29 04:26:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-0573cae2]) (logid:d55a54a9) Cleaned up 0 VM scheduled job entries 2025-06-29 04:26:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-6ab21b8b]) (logid:7a1b6426) VM Operation Thread Running 2025-06-29 04:26:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6a399985]) (logid:fd4b9345) ClusterDRS.poll is being called at 2025-06-29T02:27:00+0000 2025-06-29 04:26:48,356 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6a399985]) (logid:fd4b9345) Removed 0 old drs migration plans 2025-06-29 04:26:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2c34834]) (logid:5888c94f) No inactive management server node found 2025-06-29 04:26:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2c34834]) (logid:5888c94f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:50,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8614: Seq 7-8614: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:26:50,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8614 2025-06-29 04:26:50,417 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2de22123]) (logid:f843f71f) HA health check task is running... 2025-06-29 04:26:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b40b0b5d]) (logid:ab15795a) No inactive management server node found 2025-06-29 04:26:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b40b0b5d]) (logid:ab15795a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55158f84]) (logid:744a2b03) No inactive management server node found 2025-06-29 04:26:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55158f84]) (logid:744a2b03) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:53,151 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-d405537e]) (logid:1a515e11) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:26:53,162 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-d405537e]) (logid:1a515e11) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:26:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63587fc5]) (logid:09bec324) No inactive management server node found 2025-06-29 04:26:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63587fc5]) (logid:09bec324) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:54,419 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-54c915dc]) (logid:dfcf07fd) HA health check task is running... 2025-06-29 04:26:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92537b99]) (logid:e96d21ad) No inactive management server node found 2025-06-29 04:26:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92537b99]) (logid:e96d21ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9db6ba67]) (logid:4acfb775) No inactive management server node found 2025-06-29 04:26:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9db6ba67]) (logid:4acfb775) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f0942a6c]) (logid:c42d0e35) Enabled non-edge zones available for scan: 1 2025-06-29 04:26:58,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:26:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f0942a6c]) (logid:c42d0e35) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:26:58,095 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Seq 2-4725683383995203798: Routed from 90520738888109 2025-06-29 04:26:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f0942a6c]) (logid:c42d0e35) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:26:58,138 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Seq 1-4725683383995203798: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:26:58,138 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 04:26:58,166 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:26:58,167 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Seq 2-3926857400090362091: Routed from 90520738888109 2025-06-29 04:26:58,212 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Seq 2-3926857400090362091: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:26:58,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-077463e3]) (logid:c8221bba) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 04:26:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-85ebe46d]) (logid:d9ed04a5) Enabled non-edge zones available for scan: 1 2025-06-29 04:26:58,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-85ebe46d]) (logid:d9ed04a5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:26:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92976d1d]) (logid:923f4e94) No inactive management server node found 2025-06-29 04:26:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92976d1d]) (logid:923f4e94) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:26:58,420 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1297d10c]) (logid:97f96be7) HA health check task is running... 2025-06-29 04:26:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c9227c9]) (logid:b8be15f2) No inactive management server node found 2025-06-29 04:26:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c9227c9]) (logid:b8be15f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:00,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8615: Seq 7-8615: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:27:00,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8615 2025-06-29 04:27:01,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5400aa26]) (logid:efc76aa2) No inactive management server node found 2025-06-29 04:27:01,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5400aa26]) (logid:efc76aa2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:02,421 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-732e7103]) (logid:823980c9) HA health check task is running... 2025-06-29 04:27:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-360add4b]) (logid:000bc289) No inactive management server node found 2025-06-29 04:27:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-360add4b]) (logid:000bc289) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c859f72e]) (logid:b253279b) No inactive management server node found 2025-06-29 04:27:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c859f72e]) (logid:b253279b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6825fde]) (logid:1e755deb) No inactive management server node found 2025-06-29 04:27:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6825fde]) (logid:1e755deb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:06,422 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1cafedd9]) (logid:9cb9b2e0) HA health check task is running... 2025-06-29 04:27:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66123585]) (logid:dc55abc7) No inactive management server node found 2025-06-29 04:27:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66123585]) (logid:dc55abc7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86e2d4d4]) (logid:1081da73) No inactive management server node found 2025-06-29 04:27:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-86e2d4d4]) (logid:1081da73) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:10,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8616: Seq 7-8616: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:27:10,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8616 2025-06-29 04:27:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d282ac0]) (logid:5fa29f8a) No inactive management server node found 2025-06-29 04:27:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d282ac0]) (logid:5fa29f8a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:10,423 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e571c868]) (logid:7acaa894) HA health check task is running... 2025-06-29 04:27:11,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db5a3fb4]) (logid:4b013539) No inactive management server node found 2025-06-29 04:27:11,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db5a3fb4]) (logid:4b013539) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-39a6220e]) (logid:b18df302) Checking if any host reservation can be released ... 2025-06-29 04:27:13,010 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-39a6220e]) (logid:b18df302) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-39a6220e]) (logid:b18df302) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-39a6220e]) (logid:b18df302) Done running HostReservationReleaseChecker ... 2025-06-29 04:27:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d583afde]) (logid:1f8d0811) Found 0 routers to update status. 2025-06-29 04:27:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d583afde]) (logid:1f8d0811) Found 0 VPC's to update Redundant State. 2025-06-29 04:27:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d583afde]) (logid:1f8d0811) Found 0 networks to update RvR status. 2025-06-29 04:27:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Running Capacity Checker ... 2025-06-29 04:27:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) recalculating system capacity 2025-06-29 04:27:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Executing cpu/ram capacity update 2025-06-29 04:27:13,170 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:13,178 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:13,184 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:27:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:27:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:27:13,196 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:13,197 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:13,202 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:27:13,204 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:27:13,204 DEBUG [c.c.c.CapacityManagerImpl] (pool-32-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:27:13,213 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Done executing cpu/ram capacity update 2025-06-29 04:27:13,213 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Executing storage capacity update 2025-06-29 04:27:13,220 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:27:13,221 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:27:13,221 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:27:13,224 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:27:13,230 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:27:13,230 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:27:13,230 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:27:13,233 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:27:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:27:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:27:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:27:13,245 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:27:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:27:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:27:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:27:13,257 DEBUG [c.c.s.StorageManagerImpl] (pool-33-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:27:13,257 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Done executing storage capacity update 2025-06-29 04:27:13,257 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Executing capacity updates for public ip and Vlans 2025-06-29 04:27:13,271 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Done capacity updates for public ip and Vlans 2025-06-29 04:27:13,271 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Executing capacity updates for private ip 2025-06-29 04:27:13,274 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Done executing capacity updates for private ip 2025-06-29 04:27:13,274 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Done recalculating system capacity 2025-06-29 04:27:13,301 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-5703cec7]) (logid:86bfa621) Done running Capacity Checker ... 2025-06-29 04:27:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3e7959d]) (logid:3eb4367f) No inactive management server node found 2025-06-29 04:27:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3e7959d]) (logid:3eb4367f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:14,424 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4ef058ea]) (logid:fbdcdf29) HA health check task is running... 2025-06-29 04:27:14,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b3da8b6]) (logid:b3872b29) No inactive management server node found 2025-06-29 04:27:14,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b3da8b6]) (logid:b3872b29) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74bddf0a]) (logid:be8b43f6) No inactive management server node found 2025-06-29 04:27:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74bddf0a]) (logid:be8b43f6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f10b7b3f]) (logid:b0b1b53e) No inactive management server node found 2025-06-29 04:27:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f10b7b3f]) (logid:b0b1b53e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:18,034 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-46b1bca5]) (logid:fad4cb13) AutoScaling Monitor is running... 2025-06-29 04:27:18,101 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-929b07d2]) (logid:8bb22b21) Starting to remove all volume_stats rows older than [Sat Jun 28 16:27:18 CEST 2025]. 2025-06-29 04:27:18,102 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-929b07d2]) (logid:8bb22b21) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:27:18 CEST 2025]. 2025-06-29 04:27:18,426 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a57cce5b]) (logid:d0b48865) HA health check task is running... 2025-06-29 04:27:18,647 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-661ff070]) (logid:cbeda885) Starting to remove all vm_stats rows older than [Sat Jun 28 16:27:18 CEST 2025]. 2025-06-29 04:27:18,670 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-661ff070]) (logid:cbeda885) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:27:18 CEST 2025]. 2025-06-29 04:27:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a6f0b7c]) (logid:896d0aa8) No inactive management server node found 2025-06-29 04:27:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a6f0b7c]) (logid:896d0aa8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:20,033 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:27:20,035 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:27:20,035 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) Seq 2-4725683383995203799: Routed from 90520738888109 2025-06-29 04:27:20,079 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) Seq 1-4725683383995203799: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:27:20,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:27:20,081 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) Seq 2-3926857400090362092: Routed from 90520738888109 2025-06-29 04:27:20,126 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-73b69ec4]) (logid:758e5435) Seq 2-3926857400090362092: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:27:20,341 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8617: Seq 7-8617: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:27:20,351 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8617 2025-06-29 04:27:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98f88ee3]) (logid:fea47523) No inactive management server node found 2025-06-29 04:27:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98f88ee3]) (logid:fea47523) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf9dc7d0]) (logid:fb37f224) No inactive management server node found 2025-06-29 04:27:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf9dc7d0]) (logid:fb37f224) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:22,426 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-38db8221]) (logid:57c2fd00) HA health check task is running... 2025-06-29 04:27:23,513 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:27:23,517 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:27:23,518 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) Seq 2-4725683383995203800: Routed from 90520738888109 2025-06-29 04:27:23,571 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) Seq 1-4725683383995203800: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:27:23,605 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:27:23,606 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) Seq 2-3926857400090362093: Routed from 90520738888109 2025-06-29 04:27:23,694 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-547c1edb]) (logid:8c2b4190) Seq 2-3926857400090362093: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:27:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e98adf4]) (logid:66127c64) No inactive management server node found 2025-06-29 04:27:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e98adf4]) (logid:66127c64) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:24,422 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) StorageCollector is running... 2025-06-29 04:27:24,428 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:27:24,429 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:27:24,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:27:24,430 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 2-3938116399158788154: Routed from 90520738888109 2025-06-29 04:27:24,486 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 4-3938116399158788154: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:27:24,490 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:27:24,490 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:27:24,490 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:27:24,491 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 2-4725683383995203801: Routed from 90520738888109 2025-06-29 04:27:24,539 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 1-4725683383995203801: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:27:24,549 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:27:24,549 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:27:24,549 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:27:24,550 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 2-4725683383995203802: Routed from 90520738888109 2025-06-29 04:27:24,637 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 1-4725683383995203802: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:27:24,647 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:27:24,648 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:27:24,648 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:27:24,649 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 2-4725683383995203803: Routed from 90520738888109 2025-06-29 04:27:24,732 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 1-4725683383995203803: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:27:24,734 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:27:24,734 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:27:24,734 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:27:24,735 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 2-3926857400090362094: Routed from 90520738888109 2025-06-29 04:27:24,738 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e25e9454]) (logid:b75f6492) Seq 2-3926857400090362094: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:27:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d8898c1]) (logid:e62f2cde) No inactive management server node found 2025-06-29 04:27:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d8898c1]) (logid:e62f2cde) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:26,428 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3c878735]) (logid:736e8fd1) HA health check task is running... 2025-06-29 04:27:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb48af24]) (logid:426b9c69) No inactive management server node found 2025-06-29 04:27:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb48af24]) (logid:426b9c69) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a73ea4f0]) (logid:2feb14b0) Enabled non-edge zones available for scan: 1 2025-06-29 04:27:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a73ea4f0]) (logid:2feb14b0) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:27:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a73ea4f0]) (logid:2feb14b0) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:27:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5c6cae82]) (logid:8d7f6cc3) Enabled non-edge zones available for scan: 1 2025-06-29 04:27:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5c6cae82]) (logid:8d7f6cc3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:27:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-241eda6d]) (logid:dbb95fe0) No inactive management server node found 2025-06-29 04:27:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-241eda6d]) (logid:dbb95fe0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a63f4aa]) (logid:63341b79) No inactive management server node found 2025-06-29 04:27:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a63f4aa]) (logid:63341b79) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:30,341 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8618: Seq 7-8618: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:27:30,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8618 2025-06-29 04:27:30,429 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-566ca6eb]) (logid:56c27d54) HA health check task is running... 2025-06-29 04:27:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c5fbfec]) (logid:6880b5d0) No inactive management server node found 2025-06-29 04:27:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c5fbfec]) (logid:6880b5d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27e07b61]) (logid:36e0292c) No inactive management server node found 2025-06-29 04:27:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27e07b61]) (logid:36e0292c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0046b10a]) (logid:a3d5e02f) No inactive management server node found 2025-06-29 04:27:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0046b10a]) (logid:a3d5e02f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:34,430 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9621d5ff]) (logid:31f72110) HA health check task is running... 2025-06-29 04:27:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f21df8e]) (logid:e0c5547e) No inactive management server node found 2025-06-29 04:27:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f21df8e]) (logid:e0c5547e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d06c956c]) (logid:0fd4db99) No inactive management server node found 2025-06-29 04:27:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d06c956c]) (logid:0fd4db99) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:38,432 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a6d8a013]) (logid:b6f6e70b) HA health check task is running... 2025-06-29 04:27:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f644738a]) (logid:c3463365) No inactive management server node found 2025-06-29 04:27:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f644738a]) (logid:c3463365) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:40,342 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8619: Seq 7-8619: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:27:40,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8619 2025-06-29 04:27:40,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de1f9328]) (logid:0dd9d1f7) No inactive management server node found 2025-06-29 04:27:40,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de1f9328]) (logid:0dd9d1f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e936cd4]) (logid:3e87ab73) No inactive management server node found 2025-06-29 04:27:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e936cd4]) (logid:3e87ab73) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:42,433 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f4c3803b]) (logid:b22f9ab6) HA health check task is running... 2025-06-29 04:27:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-420e5f3e]) (logid:3c186145) DbCollector is running... 2025-06-29 04:27:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d014047]) (logid:d4e8d903) Found 0 routers to update status. 2025-06-29 04:27:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d014047]) (logid:d4e8d903) Found 0 VPC's to update Redundant State. 2025-06-29 04:27:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d014047]) (logid:d4e8d903) Found 0 networks to update RvR status. 2025-06-29 04:27:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7ec96bc]) (logid:5432a645) No inactive management server node found 2025-06-29 04:27:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7ec96bc]) (logid:5432a645) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) ManagementServerCollector is running... 2025-06-29 04:27:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) DB properties were already loaded 2025-06-29 04:27:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:27:44,055 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Execution of process [4067044] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:27:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Exit value of process [4067044] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:27:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Process [4067044] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:27:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) usage install: null 2025-06-29 04:27:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Sessions active: 0 2025-06-29 04:27:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:27:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067047] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:27:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:27:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067052] for command [/bin/bash -c uname -r ]. 2025-06-29 04:27:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:27:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067053] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:27:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) system memory from /proc: 4887515136 2025-06-29 04:27:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:27:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067058] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:27:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) free memory from /proc: 346021888 2025-06-29 04:27:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:27:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067063] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:27:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) used memory from /proc: 1024488 2025-06-29 04:27:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:27:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067064] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:27:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:27:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067065] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:27:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:27:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067066] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:27:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:27:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067073] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:27:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:27:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Successfully executed process [4067074] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail buffersdirect.count, 93 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail buffersdirect.used, 321668 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail buffersmapped.capacity, 0 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail buffersmapped.count, 0 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail buffersmapped.used, 0 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail jvmuptime, 3123829 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memoryheap.committed, 638058496 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memoryheap.init, 77594624 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memoryheap.usage, 0.1431396660389595 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorynon-heap.committed, 197853184 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorynon-heap.max, -1 2025-06-29 04:27:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorynon-heap.usage, -1.95546992E8 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorynon-heap.used, 195546992 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11619297354038922 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22641800246609126 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.Metaspace.usage, 0.9930485378204374 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.PS-Eden-Space.usage, 0.13253271073890363 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorytotal.committed, 835911680 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorytotal.init, 85262336 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorytotal.max, 1908932607 2025-06-29 04:27:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail memorytotal.used, 468790968 2025-06-29 04:27:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail threadsdeadlocks, [] 2025-06-29 04:27:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail threadsnew.count, 0 2025-06-29 04:27:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:27:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-b4e01028]) (logid:bc2a1470) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:27:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":365691008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3123759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":467914480,"totalCommitted":835911680,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":273243976,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346021888,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.07,0.26],"systemCyclesUsage":[66415625,53835835,4589806934],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:27:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 103, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:27:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":365691008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3123759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":467914480,"totalCommitted":835911680,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":273243976,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346021888,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.07,0.26],"systemCyclesUsage":[66415625,53835835,4589806934],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:27:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:27:44,137 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-23:[ctx-2bf5dcad]) (logid:737efe89) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:27:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":365691008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3123759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":467914480,"totalCommitted":835911680,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":273243976,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346021888,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.07,0.26],"systemCyclesUsage":[66415625,53835835,4589806934],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:27:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:27:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 103, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:27:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":365691008,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3123759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.02,"totalInit":85262336,"totalUsed":467914480,"totalCommitted":835911680,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":273243976,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346021888,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.02,0.07,0.26],"systemCyclesUsage":[66415625,53835835,4589806934],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:27:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6daf54db]) (logid:090f9e39) No inactive management server node found 2025-06-29 04:27:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6daf54db]) (logid:090f9e39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:27:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:27:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-966441c6]) (logid:de618cc7) No inactive management server node found 2025-06-29 04:27:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-966441c6]) (logid:de618cc7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:46,434 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ae69f47c]) (logid:55f8ad03) HA health check task is running... 2025-06-29 04:27:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:27:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:27:47,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:47,094 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:27:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:27:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdba9f62]) (logid:aa215a0a) No inactive management server node found 2025-06-29 04:27:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdba9f62]) (logid:aa215a0a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7d0e368c]) (logid:fc618016) Resetting hosts suitable for reconnect 2025-06-29 04:27:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7d0e368c]) (logid:fc618016) Completed resetting hosts suitable for reconnect 2025-06-29 04:27:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7d0e368c]) (logid:fc618016) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:27:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7d0e368c]) (logid:fc618016) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:27:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7d0e368c]) (logid:fc618016) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:27:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-7d0e368c]) (logid:fc618016) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:27:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f3a278c2]) (logid:40825532) VM scheduler.poll is being called at 2025-06-29T02:28:00+0000 2025-06-29 04:27:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f3a278c2]) (logid:40825532) Got 0 scheduled jobs to be executed at 2025-06-29T02:28:00+0000 2025-06-29 04:27:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f3a278c2]) (logid:40825532) Cleaned up 0 VM scheduled job entries 2025-06-29 04:27:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-fc9d5083]) (logid:0930dc4b) VM Operation Thread Running 2025-06-29 04:27:48,348 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1c654195]) (logid:80358e52) ClusterDRS.poll is being called at 2025-06-29T02:28:00+0000 2025-06-29 04:27:48,355 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1c654195]) (logid:80358e52) Removed 0 old drs migration plans 2025-06-29 04:27:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38cff05c]) (logid:ee756500) No inactive management server node found 2025-06-29 04:27:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38cff05c]) (logid:ee756500) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:50,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8621: Seq 7-8621: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:27:50,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8621 2025-06-29 04:27:50,435 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e05f7fe5]) (logid:a6b00401) HA health check task is running... 2025-06-29 04:27:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6837303]) (logid:a7fff8ec) No inactive management server node found 2025-06-29 04:27:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6837303]) (logid:a7fff8ec) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:52,367 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c077f55]) (logid:8891e3ec) No inactive management server node found 2025-06-29 04:27:52,367 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c077f55]) (logid:8891e3ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fd022e5]) (logid:5d06d65f) No inactive management server node found 2025-06-29 04:27:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fd022e5]) (logid:5d06d65f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:54,437 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e9121d3b]) (logid:5c62b61f) HA health check task is running... 2025-06-29 04:27:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-580f8d5d]) (logid:cf7ac8b5) No inactive management server node found 2025-06-29 04:27:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-580f8d5d]) (logid:cf7ac8b5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-378e223a]) (logid:83eef0d4) No inactive management server node found 2025-06-29 04:27:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-378e223a]) (logid:83eef0d4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a746f49f]) (logid:20d053d5) Enabled non-edge zones available for scan: 1 2025-06-29 04:27:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a746f49f]) (logid:20d053d5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:27:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a746f49f]) (logid:20d053d5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:27:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-80e01969]) (logid:e88052aa) Enabled non-edge zones available for scan: 1 2025-06-29 04:27:58,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-80e01969]) (logid:e88052aa) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:27:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6a8dcd9]) (logid:a0a22367) No inactive management server node found 2025-06-29 04:27:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6a8dcd9]) (logid:a0a22367) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:27:58,438 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f72684ee]) (logid:c1e2f625) HA health check task is running... 2025-06-29 04:27:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a271298]) (logid:98b3e916) No inactive management server node found 2025-06-29 04:27:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a271298]) (logid:98b3e916) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:00,342 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8622: Seq 7-8622: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:00,351 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8622 2025-06-29 04:28:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a619617]) (logid:9afeb50f) No inactive management server node found 2025-06-29 04:28:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a619617]) (logid:9afeb50f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:02,439 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5ae51a7b]) (logid:c7af7da1) HA health check task is running... 2025-06-29 04:28:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca9dee42]) (logid:45e84c28) No inactive management server node found 2025-06-29 04:28:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca9dee42]) (logid:45e84c28) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed127821]) (logid:52db615c) No inactive management server node found 2025-06-29 04:28:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed127821]) (logid:52db615c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1df168ce]) (logid:b16125aa) No inactive management server node found 2025-06-29 04:28:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1df168ce]) (logid:b16125aa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:06,441 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-9bb4ad43]) (logid:1026850e) HA health check task is running... 2025-06-29 04:28:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a8dd890]) (logid:4e0660e5) No inactive management server node found 2025-06-29 04:28:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a8dd890]) (logid:4e0660e5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ab8e692]) (logid:d4cf1c8c) No inactive management server node found 2025-06-29 04:28:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ab8e692]) (logid:d4cf1c8c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:10,343 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8623: Seq 7-8623: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:10,352 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8623 2025-06-29 04:28:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0197535f]) (logid:7baec317) No inactive management server node found 2025-06-29 04:28:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0197535f]) (logid:7baec317) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:10,442 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-71f7fd90]) (logid:ae34ce20) HA health check task is running... 2025-06-29 04:28:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dae69c26]) (logid:decdf315) No inactive management server node found 2025-06-29 04:28:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dae69c26]) (logid:decdf315) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ee249435]) (logid:6426f834) Found 0 routers to update status. 2025-06-29 04:28:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ee249435]) (logid:6426f834) Found 0 VPC's to update Redundant State. 2025-06-29 04:28:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ee249435]) (logid:6426f834) Found 0 networks to update RvR status. 2025-06-29 04:28:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50a22668]) (logid:0278ce33) No inactive management server node found 2025-06-29 04:28:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50a22668]) (logid:0278ce33) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:14,443 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3afd018d]) (logid:1faaf894) HA health check task is running... 2025-06-29 04:28:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a48e155]) (logid:f7ba9c1f) No inactive management server node found 2025-06-29 04:28:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a48e155]) (logid:f7ba9c1f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b0e35bc]) (logid:ccfe3f04) No inactive management server node found 2025-06-29 04:28:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b0e35bc]) (logid:ccfe3f04) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4f19136]) (logid:dfa1d2d0) No inactive management server node found 2025-06-29 04:28:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4f19136]) (logid:dfa1d2d0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:18,036 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-93f80b54]) (logid:d303821d) AutoScaling Monitor is running... 2025-06-29 04:28:18,104 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-f8c30819]) (logid:114b97f6) Starting to remove all volume_stats rows older than [Sat Jun 28 16:28:18 CEST 2025]. 2025-06-29 04:28:18,104 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-f8c30819]) (logid:114b97f6) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:28:18 CEST 2025]. 2025-06-29 04:28:18,445 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a934f155]) (logid:d9ac1528) HA health check task is running... 2025-06-29 04:28:18,671 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-542d32da]) (logid:aa09e2fd) Starting to remove all vm_stats rows older than [Sat Jun 28 16:28:18 CEST 2025]. 2025-06-29 04:28:18,694 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-542d32da]) (logid:aa09e2fd) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:28:18 CEST 2025]. 2025-06-29 04:28:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-494f932d]) (logid:4b15ae22) No inactive management server node found 2025-06-29 04:28:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-494f932d]) (logid:4b15ae22) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:20,129 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:28:20,130 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:28:20,132 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) Seq 2-4725683383995203804: Routed from 90520738888109 2025-06-29 04:28:20,175 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) Seq 1-4725683383995203804: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:28:20,176 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:28:20,177 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) Seq 2-3926857400090362095: Routed from 90520738888109 2025-06-29 04:28:20,222 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-93c817d7]) (logid:aceeea02) Seq 2-3926857400090362095: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:28:20,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8624: Seq 7-8624: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:20,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8624 2025-06-29 04:28:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f448cf00]) (logid:8921dc7f) No inactive management server node found 2025-06-29 04:28:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f448cf00]) (logid:8921dc7f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b744af8d]) (logid:50d2d25a) No inactive management server node found 2025-06-29 04:28:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b744af8d]) (logid:50d2d25a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:22,446 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f5f0a354]) (logid:09e6c6a8) HA health check task is running... 2025-06-29 04:28:23,751 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:28:23,754 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:28:23,755 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) Seq 2-4725683383995203805: Routed from 90520738888109 2025-06-29 04:28:23,808 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) Seq 1-4725683383995203805: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:28:23,841 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:28:23,842 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) Seq 2-3926857400090362096: Routed from 90520738888109 2025-06-29 04:28:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8feb9256]) (logid:a2085fdb) No inactive management server node found 2025-06-29 04:28:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8feb9256]) (logid:a2085fdb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:23,938 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-b4963985]) (logid:8de22b0a) Seq 2-3926857400090362096: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:28:24,739 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) StorageCollector is running... 2025-06-29 04:28:24,744 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:28:24,744 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:28:24,744 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:28:24,745 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-3938116399158788155: Routed from 90520738888109 2025-06-29 04:28:24,801 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 4-3938116399158788155: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:28:24,804 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:28:24,804 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:28:24,804 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:28:24,805 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-4725683383995203806: Routed from 90520738888109 2025-06-29 04:28:24,853 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 1-4725683383995203806: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:28:24,864 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:28:24,864 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:28:24,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:28:24,865 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-3926857400090362097: Routed from 90520738888109 2025-06-29 04:28:24,877 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-3926857400090362097: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:28:24,886 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:28:24,886 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:28:24,886 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:28:24,887 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-4725683383995203807: Routed from 90520738888109 2025-06-29 04:28:24,972 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 1-4725683383995203807: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:28:24,974 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:28:24,974 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:28:24,974 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:28:24,975 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-3926857400090362098: Routed from 90520738888109 2025-06-29 04:28:25,026 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-737b3459]) (logid:c8f4ae7c) Seq 2-3926857400090362098: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:28:25,343 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8625: Seq 7-8625: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:25,352 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8625 2025-06-29 04:28:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecaedf32]) (logid:e1a3dc56) No inactive management server node found 2025-06-29 04:28:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecaedf32]) (logid:e1a3dc56) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:26,447 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-dde54b0a]) (logid:0665dc31) HA health check task is running... 2025-06-29 04:28:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfa46ff7]) (logid:2f198436) No inactive management server node found 2025-06-29 04:28:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfa46ff7]) (logid:2f198436) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1b792ddd]) (logid:69e2dded) Enabled non-edge zones available for scan: 1 2025-06-29 04:28:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1b792ddd]) (logid:69e2dded) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:28:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1b792ddd]) (logid:69e2dded) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:28:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ee2fa4c6]) (logid:8ff23b16) Enabled non-edge zones available for scan: 1 2025-06-29 04:28:28,275 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ee2fa4c6]) (logid:8ff23b16) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:28:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f0bbd1d]) (logid:ce3a652f) No inactive management server node found 2025-06-29 04:28:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f0bbd1d]) (logid:ce3a652f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c2bc45b]) (logid:67c11494) No inactive management server node found 2025-06-29 04:28:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c2bc45b]) (logid:67c11494) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:30,448 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ead9a1dd]) (logid:de1dd1d6) HA health check task is running... 2025-06-29 04:28:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-924e9476]) (logid:f688e091) No inactive management server node found 2025-06-29 04:28:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-924e9476]) (logid:f688e091) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f3b31ad]) (logid:284a4e67) No inactive management server node found 2025-06-29 04:28:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f3b31ad]) (logid:284a4e67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2b0d191]) (logid:8bd1f1b6) No inactive management server node found 2025-06-29 04:28:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2b0d191]) (logid:8bd1f1b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:34,449 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-fd69f9ac]) (logid:ec2df5d3) HA health check task is running... 2025-06-29 04:28:35,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8626: Seq 7-8626: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:35,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8626 2025-06-29 04:28:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e36b9973]) (logid:e986402e) No inactive management server node found 2025-06-29 04:28:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e36b9973]) (logid:e986402e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-054d4c6e]) (logid:b9d08582) No inactive management server node found 2025-06-29 04:28:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-054d4c6e]) (logid:b9d08582) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:38,451 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8ab90cb5]) (logid:1354b27e) HA health check task is running... 2025-06-29 04:28:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-116e276d]) (logid:20e232e5) No inactive management server node found 2025-06-29 04:28:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-116e276d]) (logid:20e232e5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c037360]) (logid:1c003dfd) No inactive management server node found 2025-06-29 04:28:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c037360]) (logid:1c003dfd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:41,892 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9221b82e]) (logid:76fb1b8f) No inactive management server node found 2025-06-29 04:28:41,892 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9221b82e]) (logid:76fb1b8f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:42,452 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f67fc7a7]) (logid:f0a31182) HA health check task is running... 2025-06-29 04:28:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-5f5b7d0d]) (logid:0bd30588) DbCollector is running... 2025-06-29 04:28:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-41974cee]) (logid:a5ffd26a) Found 0 routers to update status. 2025-06-29 04:28:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-41974cee]) (logid:a5ffd26a) Found 0 VPC's to update Redundant State. 2025-06-29 04:28:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-41974cee]) (logid:a5ffd26a) Found 0 networks to update RvR status. 2025-06-29 04:28:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce26f716]) (logid:f1cc5f7d) No inactive management server node found 2025-06-29 04:28:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce26f716]) (logid:f1cc5f7d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) ManagementServerCollector is running... 2025-06-29 04:28:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) DB properties were already loaded 2025-06-29 04:28:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:28:44,057 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Execution of process [4067087] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:28:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Exit value of process [4067087] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:28:44,057 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Process [4067087] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:28:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) usage install: null 2025-06-29 04:28:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Sessions active: 0 2025-06-29 04:28:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:28:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067090] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:28:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:28:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067095] for command [/bin/bash -c uname -r ]. 2025-06-29 04:28:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:28:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067096] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:28:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) system memory from /proc: 4887515136 2025-06-29 04:28:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:28:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067101] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:28:44,084 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) free memory from /proc: 345804800 2025-06-29 04:28:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:28:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067106] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:28:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) used memory from /proc: 1024488 2025-06-29 04:28:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:28:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067107] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:28:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:28:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067108] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:28:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:28:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067109] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:28:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:28:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067116] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:28:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:28:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Successfully executed process [4067117] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail buffersdirect.count, 93 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail buffersdirect.used, 321668 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail buffersmapped.capacity, 0 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail buffersmapped.count, 0 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail buffersmapped.used, 0 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail jvmuptime, 3183836 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memoryheap.committed, 638058496 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memoryheap.init, 77594624 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memoryheap.usage, 0.15462735497470217 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorynon-heap.committed, 197984256 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorynon-heap.max, -1 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorynon-heap.usage, -1.95654536E8 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorynon-heap.used, 195654536 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:28:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11635959577446015 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22697828839937348 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.Metaspace.usage, 0.9931791633900223 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.PS-Eden-Space.usage, 0.16602084157560423 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorytotal.committed, 836042752 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorytotal.init, 85262336 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorytotal.max, 1908932607 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail memorytotal.used, 490827736 2025-06-29 04:28:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail threadsdeadlocks, [] 2025-06-29 04:28:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail threadsnew.count, 0 2025-06-29 04:28:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:28:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-c24c803e]) (logid:aa4a67d0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:28:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":343634696,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":490078336,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295173200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":345804800,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.05,0.24],"systemCyclesUsage":[66415934,53836136,4589830079],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:28:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 105, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:28:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":343634696,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":490078336,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295173200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":345804800,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.05,0.24],"systemCyclesUsage":[66415934,53836136,4589830079],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:28:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:28:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:28:44,144 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 105, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:28:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":343634696,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":490078336,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295173200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":345804800,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.05,0.24],"systemCyclesUsage":[66415934,53836136,4589830079],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:28:44,144 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-24:[ctx-43036694]) (logid:c14b7833) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:28:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":343634696,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3183761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":490078336,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":295173200,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":345804800,"systemMemoryUsed":1024488,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.05,0.24],"systemCyclesUsage":[66415934,53836136,4589830079],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:28:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-751f811f]) (logid:9af22f27) No inactive management server node found 2025-06-29 04:28:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-751f811f]) (logid:9af22f27) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:45,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8627: Seq 7-8627: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:45,355 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8627 2025-06-29 04:28:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:28:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:28:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3748c5ea]) (logid:8856171c) No inactive management server node found 2025-06-29 04:28:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3748c5ea]) (logid:8856171c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:46,453 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8e36094e]) (logid:9aa14e09) HA health check task is running... 2025-06-29 04:28:46,648 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:28:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:28:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:28:46,654 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:28:47,079 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:28:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:28:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:28:47,083 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:28:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5faa844]) (logid:a4db6215) No inactive management server node found 2025-06-29 04:28:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5faa844]) (logid:a4db6215) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e1d691c2]) (logid:2dbfaf4f) VM scheduler.poll is being called at 2025-06-29T02:29:00+0000 2025-06-29 04:28:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e1d691c2]) (logid:2dbfaf4f) Got 0 scheduled jobs to be executed at 2025-06-29T02:29:00+0000 2025-06-29 04:28:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-e1d691c2]) (logid:2dbfaf4f) Cleaned up 0 VM scheduled job entries 2025-06-29 04:28:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-2e298633]) (logid:05aa0dfc) VM Operation Thread Running 2025-06-29 04:28:48,349 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a00e20f2]) (logid:ac2ba2e0) ClusterDRS.poll is being called at 2025-06-29T02:29:00+0000 2025-06-29 04:28:48,354 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a00e20f2]) (logid:ac2ba2e0) Removed 0 old drs migration plans 2025-06-29 04:28:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7224f23]) (logid:fd4c629a) No inactive management server node found 2025-06-29 04:28:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7224f23]) (logid:fd4c629a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:50,455 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b3e1252f]) (logid:e63a76bc) HA health check task is running... 2025-06-29 04:28:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0721ed74]) (logid:08238013) No inactive management server node found 2025-06-29 04:28:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0721ed74]) (logid:08238013) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ad9153b]) (logid:e78b404b) No inactive management server node found 2025-06-29 04:28:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ad9153b]) (logid:e78b404b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e8097f0]) (logid:fc220c4e) No inactive management server node found 2025-06-29 04:28:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e8097f0]) (logid:fc220c4e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:54,456 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-fa245aa3]) (logid:1b06d6bb) HA health check task is running... 2025-06-29 04:28:55,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8629: Seq 7-8629: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:28:55,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8629 2025-06-29 04:28:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6525b9b]) (logid:1d638e5c) No inactive management server node found 2025-06-29 04:28:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6525b9b]) (logid:1d638e5c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-452c7a6f]) (logid:7a8365c6) No inactive management server node found 2025-06-29 04:28:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-452c7a6f]) (logid:7a8365c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1746593a]) (logid:212ddc40) Enabled non-edge zones available for scan: 1 2025-06-29 04:28:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-1746593a]) (logid:212ddc40) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:28:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-1746593a]) (logid:212ddc40) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:28:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1760372e]) (logid:e79b543e) Enabled non-edge zones available for scan: 1 2025-06-29 04:28:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1760372e]) (logid:e79b543e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:28:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a781bdd]) (logid:cb447e5c) No inactive management server node found 2025-06-29 04:28:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a781bdd]) (logid:cb447e5c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:28:58,457 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-fb425408]) (logid:6af1e8ad) HA health check task is running... 2025-06-29 04:28:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-476efc02]) (logid:1a6f693c) No inactive management server node found 2025-06-29 04:28:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-476efc02]) (logid:1a6f693c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3ddc144]) (logid:d6a6a5a6) No inactive management server node found 2025-06-29 04:29:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3ddc144]) (logid:d6a6a5a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:02,458 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1f6b2c87]) (logid:48fcda90) HA health check task is running... 2025-06-29 04:29:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-753657f8]) (logid:32876bad) No inactive management server node found 2025-06-29 04:29:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-753657f8]) (logid:32876bad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb713a68]) (logid:52f89528) No inactive management server node found 2025-06-29 04:29:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb713a68]) (logid:52f89528) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:05,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8630: Seq 7-8630: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:29:05,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8630 2025-06-29 04:29:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cb56d83]) (logid:eb76c736) No inactive management server node found 2025-06-29 04:29:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cb56d83]) (logid:eb76c736) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:06,460 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-379c3402]) (logid:d06b0f74) HA health check task is running... 2025-06-29 04:29:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0d03d78]) (logid:ba5698f5) No inactive management server node found 2025-06-29 04:29:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0d03d78]) (logid:ba5698f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cffd2fd]) (logid:5db3a1a1) No inactive management server node found 2025-06-29 04:29:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5cffd2fd]) (logid:5db3a1a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bb62040]) (logid:090e0564) No inactive management server node found 2025-06-29 04:29:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4bb62040]) (logid:090e0564) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:10,461 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-33e4de52]) (logid:7585ebf5) HA health check task is running... 2025-06-29 04:29:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a281ea2]) (logid:dfe0e5fd) No inactive management server node found 2025-06-29 04:29:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a281ea2]) (logid:dfe0e5fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ecabaf1e]) (logid:dbc54856) Found 0 routers to update status. 2025-06-29 04:29:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ecabaf1e]) (logid:dbc54856) Found 0 VPC's to update Redundant State. 2025-06-29 04:29:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ecabaf1e]) (logid:dbc54856) Found 0 networks to update RvR status. 2025-06-29 04:29:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43c902e5]) (logid:60ef3dad) No inactive management server node found 2025-06-29 04:29:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43c902e5]) (logid:60ef3dad) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:14,462 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b8227630]) (logid:e67f399a) HA health check task is running... 2025-06-29 04:29:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92407c96]) (logid:06b59a5b) No inactive management server node found 2025-06-29 04:29:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92407c96]) (logid:06b59a5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:15,348 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8631: Seq 7-8631: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:29:15,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8631 2025-06-29 04:29:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5030d177]) (logid:2c0fe0a1) No inactive management server node found 2025-06-29 04:29:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5030d177]) (logid:2c0fe0a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20dbe952]) (logid:5baebae0) No inactive management server node found 2025-06-29 04:29:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20dbe952]) (logid:5baebae0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:18,038 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-1a0f3544]) (logid:eadaf376) AutoScaling Monitor is running... 2025-06-29 04:29:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4080e84]) (logid:ab7fd9ce) Resetting hosts suitable for reconnect 2025-06-29 04:29:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4080e84]) (logid:ab7fd9ce) Completed resetting hosts suitable for reconnect 2025-06-29 04:29:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4080e84]) (logid:ab7fd9ce) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:29:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4080e84]) (logid:ab7fd9ce) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:29:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4080e84]) (logid:ab7fd9ce) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:29:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a4080e84]) (logid:ab7fd9ce) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:29:18,106 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-94da0f10]) (logid:af635af2) Starting to remove all volume_stats rows older than [Sat Jun 28 16:29:18 CEST 2025]. 2025-06-29 04:29:18,107 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-94da0f10]) (logid:af635af2) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:29:18 CEST 2025]. 2025-06-29 04:29:18,464 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-7492d92f]) (logid:b08aaa5e) HA health check task is running... 2025-06-29 04:29:18,695 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-0f44c7e9]) (logid:b27159c9) Starting to remove all vm_stats rows older than [Sat Jun 28 16:29:18 CEST 2025]. 2025-06-29 04:29:18,718 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-0f44c7e9]) (logid:b27159c9) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:29:18 CEST 2025]. 2025-06-29 04:29:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd451545]) (logid:206a4333) No inactive management server node found 2025-06-29 04:29:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd451545]) (logid:206a4333) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:20,225 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:29:20,228 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:29:20,231 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) Seq 2-4725683383995203808: Routed from 90520738888109 2025-06-29 04:29:20,274 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) Seq 1-4725683383995203808: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:29:20,276 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:29:20,277 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) Seq 2-3926857400090362099: Routed from 90520738888109 2025-06-29 04:29:20,322 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-f8ed0b41]) (logid:ed5c5fb4) Seq 2-3926857400090362099: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:29:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f02f5cef]) (logid:6ee78674) No inactive management server node found 2025-06-29 04:29:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f02f5cef]) (logid:6ee78674) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:29:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53121b41]) (logid:3be787a6) No inactive management server node found 2025-06-29 04:29:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-53121b41]) (logid:3be787a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:29:22,465 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-abd3130c]) (logid:74090ad6) HA health check task is running... 2025-06-29 04:29:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af6c29a3]) (logid:b79a5a27) No inactive management server node found 2025-06-29 04:29:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af6c29a3]) (logid:b79a5a27) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:29:23,992 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:29:23,994 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:29:23,995 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) Seq 2-4725683383995203809: Routed from 90520738888109 2025-06-29 04:29:24,054 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) Seq 1-4725683383995203809: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:29:24,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:29:24,088 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) Seq 2-3926857400090362100: Routed from 90520738888109 2025-06-29 04:29:24,197 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d0b1c7ef]) (logid:040033db) Seq 2-3926857400090362100: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:29:25,026 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) StorageCollector is running... 2025-06-29 04:29:25,033 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:29:25,033 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:29:25,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:29:25,035 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-3938116399158788156: Routed from 90520738888109 2025-06-29 04:29:25,142 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 4-3938116399158788156: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:29:25,146 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:29:25,146 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:29:25,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:29:25,147 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-4725683383995203810: Routed from 90520738888109 2025-06-29 04:29:25,194 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 1-4725683383995203810: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:29:25,204 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:29:25,204 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:29:25,204 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:29:25,205 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-3926857400090362101: Routed from 90520738888109 2025-06-29 04:29:25,229 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-3926857400090362101: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:29:25,240 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:29:25,240 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:29:25,240 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:29:25,241 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-4725683383995203811: Routed from 90520738888109 2025-06-29 04:29:25,285 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 1-4725683383995203811: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:29:25,287 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:29:25,287 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:29:25,287 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:29:25,288 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-3926857400090362102: Routed from 90520738888109 2025-06-29 04:29:25,332 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-4ab67edb]) (logid:e842026b) Seq 2-3926857400090362102: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:29:25,346 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8632: Seq 7-8632: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:29:25,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8632 2025-06-29 04:29:25,363 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a242b9d]) (logid:93ce4f46) No inactive management server node found 2025-06-29 04:29:25,363 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a242b9d]) (logid:93ce4f46) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:26,466 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2a0d5ec7]) (logid:462cb5c7) HA health check task is running... 2025-06-29 04:29:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-412438bb]) (logid:5e10933c) No inactive management server node found 2025-06-29 04:29:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-412438bb]) (logid:5e10933c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-23abc918]) (logid:957c5cdd) Enabled non-edge zones available for scan: 1 2025-06-29 04:29:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-23abc918]) (logid:957c5cdd) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:29:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-23abc918]) (logid:957c5cdd) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:29:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c686e2d1]) (logid:43cd01fe) Enabled non-edge zones available for scan: 1 2025-06-29 04:29:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c686e2d1]) (logid:43cd01fe) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:29:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31e35edf]) (logid:a8577d73) No inactive management server node found 2025-06-29 04:29:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31e35edf]) (logid:a8577d73) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:29,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c67dc65]) (logid:8738bd99) No inactive management server node found 2025-06-29 04:29:29,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c67dc65]) (logid:8738bd99) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:30,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8633: Seq 7-8633: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:29:30,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8633 2025-06-29 04:29:30,468 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-efaec528]) (logid:a8443820) HA health check task is running... 2025-06-29 04:29:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2e6c585]) (logid:4bd81cc4) No inactive management server node found 2025-06-29 04:29:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2e6c585]) (logid:4bd81cc4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa21ab8b]) (logid:3e6c61f8) No inactive management server node found 2025-06-29 04:29:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa21ab8b]) (logid:3e6c61f8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2977bae]) (logid:9b657934) No inactive management server node found 2025-06-29 04:29:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e2977bae]) (logid:9b657934) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:34,469 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e3aa2ba4]) (logid:9f661d03) HA health check task is running... 2025-06-29 04:29:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34a62a60]) (logid:ebb4c662) No inactive management server node found 2025-06-29 04:29:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34a62a60]) (logid:ebb4c662) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e35f4aa7]) (logid:e90d4f0b) No inactive management server node found 2025-06-29 04:29:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e35f4aa7]) (logid:e90d4f0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:29:38,470 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e2325ca9]) (logid:96a65b79) HA health check task is running... 2025-06-29 04:29:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4733b220]) (logid:617decff) No inactive management server node found 2025-06-29 04:29:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4733b220]) (logid:617decff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:40,347 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8634: Seq 7-8634: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:29:40,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8634 2025-06-29 04:29:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b194d32b]) (logid:d97fb6c6) No inactive management server node found 2025-06-29 04:29:40,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b194d32b]) (logid:d97fb6c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be9c6199]) (logid:5009d5eb) No inactive management server node found 2025-06-29 04:29:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-be9c6199]) (logid:5009d5eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:42,471 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-26865a9e]) (logid:0d9ea5a0) HA health check task is running... 2025-06-29 04:29:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-56c067d5]) (logid:d10a5f72) DbCollector is running... 2025-06-29 04:29:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-79722149]) (logid:482a1758) Found 0 routers to update status. 2025-06-29 04:29:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-79722149]) (logid:482a1758) Found 0 VPC's to update Redundant State. 2025-06-29 04:29:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-79722149]) (logid:482a1758) Found 0 networks to update RvR status. 2025-06-29 04:29:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a3f667d]) (logid:8f697f56) No inactive management server node found 2025-06-29 04:29:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a3f667d]) (logid:8f697f56) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) ManagementServerCollector is running... 2025-06-29 04:29:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) DB properties were already loaded 2025-06-29 04:29:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:29:44,058 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Execution of process [4067123] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:29:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Exit value of process [4067123] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:29:44,058 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Process [4067123] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:29:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) usage install: null 2025-06-29 04:29:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Sessions active: 0 2025-06-29 04:29:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:29:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067126] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:29:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:29:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067131] for command [/bin/bash -c uname -r ]. 2025-06-29 04:29:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:29:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067132] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:29:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) system memory from /proc: 4887515136 2025-06-29 04:29:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:29:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067137] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:29:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) free memory from /proc: 343502848 2025-06-29 04:29:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:29:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067142] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:29:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) used memory from /proc: 1024616 2025-06-29 04:29:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:29:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067143] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:29:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:29:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067144] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:29:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:29:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067145] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:29:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:29:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067152] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:29:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:29:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Successfully executed process [4067153] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail buffersdirect.count, 93 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail buffersdirect.used, 321668 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail buffersmapped.capacity, 0 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail buffersmapped.count, 0 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail buffersmapped.used, 0 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:29:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail jvmuptime, 3243834 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memoryheap.committed, 638058496 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memoryheap.init, 77594624 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memoryheap.usage, 0.16665139809901555 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorynon-heap.committed, 197984256 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorynon-heap.max, -1 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorynon-heap.usage, -1.95688176E8 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorynon-heap.used, 195688176 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1165199696747534 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22707201652947645 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.Metaspace.usage, 0.9931964461576905 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.PS-Eden-Space.usage, 0.20107251572742568 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorytotal.committed, 836042752 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorytotal.init, 85262336 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorytotal.max, 1908932607 2025-06-29 04:29:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail memorytotal.used, 513814464 2025-06-29 04:29:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail threadsdeadlocks, [] 2025-06-29 04:29:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail threadsnew.count, 0 2025-06-29 04:29:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:29:44,134 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-2a4c6c70]) (logid:802c34da) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:29:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":320701264,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3243761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513045408,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":318126288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":343502848,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.23],"systemCyclesUsage":[66416252,53836449,4589853204],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:29:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 107, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:29:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":320701264,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3243761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513045408,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":318126288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":343502848,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.23],"systemCyclesUsage":[66416252,53836449,4589853204],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:29:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:29:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-24:[ctx-8f34dc31]) (logid:a84db4cc) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:29:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":320701264,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3243761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513045408,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":318126288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":343502848,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.23],"systemCyclesUsage":[66416252,53836449,4589853204],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:29:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:29:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 107, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:29:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":320701264,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3243761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513045408,"totalCommitted":836042752,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":318126288,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":343502848,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.04,0.23],"systemCyclesUsage":[66416252,53836449,4589853204],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:29:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ead54c5]) (logid:643527c6) No inactive management server node found 2025-06-29 04:29:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ead54c5]) (logid:643527c6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:29:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:29:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f28b153]) (logid:e4fa4db3) No inactive management server node found 2025-06-29 04:29:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f28b153]) (logid:e4fa4db3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:46,473 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-05b8a1bd]) (logid:4ff85c1b) HA health check task is running... 2025-06-29 04:29:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:29:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:29:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:29:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:29:47,078 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:29:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:29:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:29:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:29:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9c1a3e4]) (logid:6330069f) No inactive management server node found 2025-06-29 04:29:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9c1a3e4]) (logid:6330069f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a43950ad]) (logid:b71b6922) VM scheduler.poll is being called at 2025-06-29T02:30:00+0000 2025-06-29 04:29:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a43950ad]) (logid:b71b6922) Got 0 scheduled jobs to be executed at 2025-06-29T02:30:00+0000 2025-06-29 04:29:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-a43950ad]) (logid:b71b6922) Cleaned up 0 VM scheduled job entries 2025-06-29 04:29:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-047ba121]) (logid:ad73d2bc) VM Operation Thread Running 2025-06-29 04:29:48,349 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5cccf6ee]) (logid:7aa0afee) ClusterDRS.poll is being called at 2025-06-29T02:30:00+0000 2025-06-29 04:29:48,355 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5cccf6ee]) (logid:7aa0afee) Removed 0 old drs migration plans 2025-06-29 04:29:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5023e549]) (logid:1ca0fadd) No inactive management server node found 2025-06-29 04:29:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5023e549]) (logid:1ca0fadd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:50,348 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8636: Seq 7-8636: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:29:50,358 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8636 2025-06-29 04:29:50,474 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-708b8832]) (logid:52f810d8) HA health check task is running... 2025-06-29 04:29:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89129f33]) (logid:d7c19fd6) No inactive management server node found 2025-06-29 04:29:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89129f33]) (logid:d7c19fd6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:52,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bda704fb]) (logid:e2aa3002) No inactive management server node found 2025-06-29 04:29:52,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bda704fb]) (logid:e2aa3002) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbacdab9]) (logid:af117162) No inactive management server node found 2025-06-29 04:29:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbacdab9]) (logid:af117162) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:54,475 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2c1488ad]) (logid:e0c495d2) HA health check task is running... 2025-06-29 04:29:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1db59b70]) (logid:740d36b0) No inactive management server node found 2025-06-29 04:29:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1db59b70]) (logid:740d36b0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03745e89]) (logid:a495bca5) No inactive management server node found 2025-06-29 04:29:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03745e89]) (logid:a495bca5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fc43d7ee]) (logid:6f912905) Enabled non-edge zones available for scan: 1 2025-06-29 04:29:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-fc43d7ee]) (logid:6f912905) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:29:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fc43d7ee]) (logid:6f912905) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:29:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-bcc3775f]) (logid:07ec49a4) Enabled non-edge zones available for scan: 1 2025-06-29 04:29:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-bcc3775f]) (logid:07ec49a4) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:29:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83dad75b]) (logid:9ed2e5e2) No inactive management server node found 2025-06-29 04:29:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83dad75b]) (logid:9ed2e5e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:29:58,477 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ce0a7b54]) (logid:61b65fd9) HA health check task is running... 2025-06-29 04:29:59,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74075fe6]) (logid:785c59ce) No inactive management server node found 2025-06-29 04:29:59,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74075fe6]) (logid:785c59ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:00,348 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8637: Seq 7-8637: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:00,360 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8637 2025-06-29 04:30:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c4b7baa]) (logid:9dfc23ed) No inactive management server node found 2025-06-29 04:30:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c4b7baa]) (logid:9dfc23ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:02,478 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8b98dba2]) (logid:a667ce85) HA health check task is running... 2025-06-29 04:30:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f3ae54d]) (logid:d7c1e886) No inactive management server node found 2025-06-29 04:30:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1f3ae54d]) (logid:d7c1e886) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a04afd95]) (logid:7c284d11) No inactive management server node found 2025-06-29 04:30:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a04afd95]) (logid:7c284d11) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c40c1856]) (logid:bc4e5508) No inactive management server node found 2025-06-29 04:30:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c40c1856]) (logid:bc4e5508) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:06,479 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2c34ef6b]) (logid:2b10cced) HA health check task is running... 2025-06-29 04:30:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb0a42e3]) (logid:ad649876) No inactive management server node found 2025-06-29 04:30:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb0a42e3]) (logid:ad649876) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d46ecfda]) (logid:a761dbda) No inactive management server node found 2025-06-29 04:30:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d46ecfda]) (logid:a761dbda) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:10,349 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8638: Seq 7-8638: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a2bfbbd]) (logid:31428d8d) No inactive management server node found 2025-06-29 04:30:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a2bfbbd]) (logid:31428d8d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:10,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8638 2025-06-29 04:30:10,480 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-faa730a8]) (logid:0f2fe77a) HA health check task is running... 2025-06-29 04:30:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0783c676]) (logid:7ed9ed6f) No inactive management server node found 2025-06-29 04:30:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0783c676]) (logid:7ed9ed6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9fed2696]) (logid:498aeca4) Found 0 routers to update status. 2025-06-29 04:30:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9fed2696]) (logid:498aeca4) Found 0 VPC's to update Redundant State. 2025-06-29 04:30:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-9fed2696]) (logid:498aeca4) Found 0 networks to update RvR status. 2025-06-29 04:30:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58a33ef5]) (logid:4abac3bd) No inactive management server node found 2025-06-29 04:30:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58a33ef5]) (logid:4abac3bd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:14,481 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-53e824f3]) (logid:0c2556fb) HA health check task is running... 2025-06-29 04:30:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97dacd0a]) (logid:e94bdcb1) No inactive management server node found 2025-06-29 04:30:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97dacd0a]) (logid:e94bdcb1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96eddf1e]) (logid:270be906) No inactive management server node found 2025-06-29 04:30:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96eddf1e]) (logid:270be906) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2109c9de]) (logid:1d442a67) No inactive management server node found 2025-06-29 04:30:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2109c9de]) (logid:1d442a67) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:18,041 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-53747bf2]) (logid:e9a47c6c) AutoScaling Monitor is running... 2025-06-29 04:30:18,109 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-b5be9b7f]) (logid:9cd1d140) Starting to remove all volume_stats rows older than [Sat Jun 28 16:30:18 CEST 2025]. 2025-06-29 04:30:18,110 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-b5be9b7f]) (logid:9cd1d140) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:30:18 CEST 2025]. 2025-06-29 04:30:18,483 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-42190d22]) (logid:bf8b0435) HA health check task is running... 2025-06-29 04:30:18,719 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-bf16c07b]) (logid:fa4ef2c9) Starting to remove all vm_stats rows older than [Sat Jun 28 16:30:18 CEST 2025]. 2025-06-29 04:30:18,742 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-bf16c07b]) (logid:fa4ef2c9) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:30:18 CEST 2025]. 2025-06-29 04:30:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1be358a9]) (logid:02253f43) No inactive management server node found 2025-06-29 04:30:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1be358a9]) (logid:02253f43) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:20,325 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:30:20,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:30:20,331 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) Seq 2-4725683383995203812: Routed from 90520738888109 2025-06-29 04:30:20,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8639: Seq 7-8639: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:20,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8639 2025-06-29 04:30:20,374 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) Seq 1-4725683383995203812: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:30:20,376 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:30:20,377 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) Seq 2-3926857400090362103: Routed from 90520738888109 2025-06-29 04:30:20,421 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-f34ca3ee]) (logid:9c6a2087) Seq 2-3926857400090362103: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:30:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d43bb7a]) (logid:fc64d51b) No inactive management server node found 2025-06-29 04:30:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d43bb7a]) (logid:fc64d51b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee827dc3]) (logid:3ff585f7) No inactive management server node found 2025-06-29 04:30:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee827dc3]) (logid:3ff585f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:22,484 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-23bb0e68]) (logid:0b382ef7) HA health check task is running... 2025-06-29 04:30:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dca65b9]) (logid:718194b5) No inactive management server node found 2025-06-29 04:30:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dca65b9]) (logid:718194b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:24,251 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:30:24,254 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:30:24,254 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) Seq 2-4725683383995203813: Routed from 90520738888109 2025-06-29 04:30:24,310 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) Seq 1-4725683383995203813: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:30:24,348 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:30:24,349 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) Seq 2-3926857400090362104: Routed from 90520738888109 2025-06-29 04:30:24,445 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c0ba14ae]) (logid:02feaeff) Seq 2-3926857400090362104: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:30:25,333 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) StorageCollector is running... 2025-06-29 04:30:25,340 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:30:25,340 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:30:25,340 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:30:25,341 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3938116399158788157: Routed from 90520738888109 2025-06-29 04:30:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f67ea4ce]) (logid:add6d54f) No inactive management server node found 2025-06-29 04:30:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f67ea4ce]) (logid:add6d54f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:25,420 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 4-3938116399158788157: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:30:25,423 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:30:25,424 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:30:25,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:30:25,425 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3926857400090362105: Routed from 90520738888109 2025-06-29 04:30:25,438 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3926857400090362105: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:30:25,451 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:30:25,451 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:30:25,451 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:30:25,452 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3926857400090362106: Routed from 90520738888109 2025-06-29 04:30:25,544 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3926857400090362106: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:30:25,557 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:30:25,557 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:30:25,557 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:30:25,558 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-4725683383995203814: Routed from 90520738888109 2025-06-29 04:30:25,602 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 1-4725683383995203814: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:30:25,604 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:30:25,604 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:30:25,604 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:30:25,605 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3926857400090362107: Routed from 90520738888109 2025-06-29 04:30:25,650 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c39a022]) (logid:43af6ed6) Seq 2-3926857400090362107: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:30:26,485 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-24674fea]) (logid:73e44767) HA health check task is running... 2025-06-29 04:30:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-095cae5b]) (logid:954dc6a3) No inactive management server node found 2025-06-29 04:30:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-095cae5b]) (logid:954dc6a3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9a470c1d]) (logid:c5595ad5) Enabled non-edge zones available for scan: 1 2025-06-29 04:30:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9a470c1d]) (logid:c5595ad5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:30:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9a470c1d]) (logid:c5595ad5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:30:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c35b4242]) (logid:0b28582e) Enabled non-edge zones available for scan: 1 2025-06-29 04:30:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c35b4242]) (logid:0b28582e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:30:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c04cee3a]) (logid:57cad96b) No inactive management server node found 2025-06-29 04:30:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c04cee3a]) (logid:57cad96b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b76502b]) (logid:4098b8ee) No inactive management server node found 2025-06-29 04:30:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b76502b]) (logid:4098b8ee) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:30,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8640: Seq 7-8640: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:30,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8640 2025-06-29 04:30:30,486 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b99d9f6e]) (logid:5c197d5d) HA health check task is running... 2025-06-29 04:30:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b84f7da]) (logid:e418be8b) No inactive management server node found 2025-06-29 04:30:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b84f7da]) (logid:e418be8b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ed98d14]) (logid:4a907797) No inactive management server node found 2025-06-29 04:30:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3ed98d14]) (logid:4a907797) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74026b1]) (logid:145fa28e) No inactive management server node found 2025-06-29 04:30:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a74026b1]) (logid:145fa28e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:34,488 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d2fce12a]) (logid:a31ed1e1) HA health check task is running... 2025-06-29 04:30:35,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8641: Seq 7-8641: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:35,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8641 2025-06-29 04:30:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88d6c8fa]) (logid:814feb11) No inactive management server node found 2025-06-29 04:30:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88d6c8fa]) (logid:814feb11) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-809a24bc]) (logid:25fc0a3b) No inactive management server node found 2025-06-29 04:30:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-809a24bc]) (logid:25fc0a3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:38,489 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d0e675ef]) (logid:385d75a0) HA health check task is running... 2025-06-29 04:30:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-240915d9]) (logid:d43cfe09) No inactive management server node found 2025-06-29 04:30:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-240915d9]) (logid:d43cfe09) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-845251d8]) (logid:a4193995) No inactive management server node found 2025-06-29 04:30:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-845251d8]) (logid:a4193995) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc3089fe]) (logid:45a4ef1e) No inactive management server node found 2025-06-29 04:30:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc3089fe]) (logid:45a4ef1e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:42,491 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-613bbff8]) (logid:09e3997e) HA health check task is running... 2025-06-29 04:30:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-1d2a918f]) (logid:6b919bc7) DbCollector is running... 2025-06-29 04:30:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3ba74413]) (logid:488cfefd) Found 0 routers to update status. 2025-06-29 04:30:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3ba74413]) (logid:488cfefd) Found 0 VPC's to update Redundant State. 2025-06-29 04:30:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3ba74413]) (logid:488cfefd) Found 0 networks to update RvR status. 2025-06-29 04:30:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74e6d64a]) (logid:cf12eaa2) No inactive management server node found 2025-06-29 04:30:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74e6d64a]) (logid:cf12eaa2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) ManagementServerCollector is running... 2025-06-29 04:30:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) DB properties were already loaded 2025-06-29 04:30:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:30:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Execution of process [4067166] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:30:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Exit value of process [4067166] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:30:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Process [4067166] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:30:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) usage install: null 2025-06-29 04:30:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Sessions active: 0 2025-06-29 04:30:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:30:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067169] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:30:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:30:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067174] for command [/bin/bash -c uname -r ]. 2025-06-29 04:30:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:30:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067175] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:30:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) system memory from /proc: 4887515136 2025-06-29 04:30:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:30:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067180] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:30:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) free memory from /proc: 344346624 2025-06-29 04:30:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:30:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067185] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:30:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) used memory from /proc: 1024616 2025-06-29 04:30:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:30:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067186] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:30:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:30:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067187] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:30:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:30:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067188] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:30:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:30:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067195] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:30:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:30:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Successfully executed process [4067196] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:30:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:30:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail buffersdirect.count, 93 2025-06-29 04:30:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail buffersdirect.used, 321668 2025-06-29 04:30:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail buffersmapped.capacity, 0 2025-06-29 04:30:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail buffersmapped.count, 0 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail buffersmapped.used, 0 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail jvmuptime, 3303833 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memoryheap.committed, 638058496 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memoryheap.init, 77594624 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memoryheap.usage, 0.1783906537993404 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorynon-heap.committed, 198115328 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorynon-heap.max, -1 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorynon-heap.usage, -1.95794496E8 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorynon-heap.used, 195794496 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11697713942948547 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22745838471023427 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.Metaspace.usage, 0.9932153940425094 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.PS-Eden-Space.usage, 0.23561612962244796 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorytotal.committed, 836173824 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorytotal.init, 85262336 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorytotal.max, 1908932607 2025-06-29 04:30:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail memorytotal.used, 536541176 2025-06-29 04:30:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail threadsdeadlocks, [] 2025-06-29 04:30:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail threadsnew.count, 0 2025-06-29 04:30:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:30:44,134 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-d6f3ecb4]) (logid:fc2b3037) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:30:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":298204784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3303760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":535648208,"totalCommitted":836173824,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":340535736,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344346624,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.21],"systemCyclesUsage":[66416592,53836759,4589876307],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:30:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 109, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:30:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":298204784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3303760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":535648208,"totalCommitted":836173824,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":340535736,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344346624,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.21],"systemCyclesUsage":[66416592,53836759,4589876307],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:30:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:30:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-24:[ctx-626bdb29]) (logid:792c21da) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:30:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":298204784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3303760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":535648208,"totalCommitted":836173824,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":340535736,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344346624,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.21],"systemCyclesUsage":[66416592,53836759,4589876307],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:30:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:30:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 109, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:30:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":298204784,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3303760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.04,"totalInit":85262336,"totalUsed":535648208,"totalCommitted":836173824,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":340535736,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":344346624,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.04,0.04,0.21],"systemCyclesUsage":[66416592,53836759,4589876307],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:30:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75bc2b6c]) (logid:737a7a07) No inactive management server node found 2025-06-29 04:30:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75bc2b6c]) (logid:737a7a07) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:30:45,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8642: Seq 7-8642: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:45,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8642 2025-06-29 04:30:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:30:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:30:46,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1495d51]) (logid:c02a1866) No inactive management server node found 2025-06-29 04:30:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1495d51]) (logid:c02a1866) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:46,492 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e92e7504]) (logid:36e755c1) HA health check task is running... 2025-06-29 04:30:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:30:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:30:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:30:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:30:47,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:30:47,091 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:30:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:30:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:30:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b2c027e]) (logid:1b3762cb) No inactive management server node found 2025-06-29 04:30:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b2c027e]) (logid:1b3762cb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ab302261]) (logid:9ebd18b8) Resetting hosts suitable for reconnect 2025-06-29 04:30:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ab302261]) (logid:9ebd18b8) Completed resetting hosts suitable for reconnect 2025-06-29 04:30:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ab302261]) (logid:9ebd18b8) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:30:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ab302261]) (logid:9ebd18b8) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:30:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ab302261]) (logid:9ebd18b8) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:30:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-ab302261]) (logid:9ebd18b8) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:30:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-965234af]) (logid:569f5277) VM scheduler.poll is being called at 2025-06-29T02:31:00+0000 2025-06-29 04:30:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-965234af]) (logid:569f5277) Got 0 scheduled jobs to be executed at 2025-06-29T02:31:00+0000 2025-06-29 04:30:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-965234af]) (logid:569f5277) Cleaned up 0 VM scheduled job entries 2025-06-29 04:30:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d0f4e118]) (logid:678a43da) VM Operation Thread Running 2025-06-29 04:30:48,349 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a7810733]) (logid:9e2b339e) ClusterDRS.poll is being called at 2025-06-29T02:31:00+0000 2025-06-29 04:30:48,356 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-a7810733]) (logid:9e2b339e) Removed 0 old drs migration plans 2025-06-29 04:30:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f2ca5b7]) (logid:6b06e514) No inactive management server node found 2025-06-29 04:30:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f2ca5b7]) (logid:6b06e514) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:50,494 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-81e6c8b3]) (logid:58f13f5f) HA health check task is running... 2025-06-29 04:30:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ee8a3bd]) (logid:e8f9b23b) No inactive management server node found 2025-06-29 04:30:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ee8a3bd]) (logid:e8f9b23b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67c5e954]) (logid:96926d44) No inactive management server node found 2025-06-29 04:30:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67c5e954]) (logid:96926d44) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35063f33]) (logid:1b975a15) No inactive management server node found 2025-06-29 04:30:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35063f33]) (logid:1b975a15) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:54,495 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-415e1181]) (logid:0f87a45b) HA health check task is running... 2025-06-29 04:30:55,350 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8644: Seq 7-8644: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:30:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df8d0342]) (logid:0032379f) No inactive management server node found 2025-06-29 04:30:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df8d0342]) (logid:0032379f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:55,362 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8644 2025-06-29 04:30:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c18f6724]) (logid:826be0fd) No inactive management server node found 2025-06-29 04:30:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c18f6724]) (logid:826be0fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-092a5612]) (logid:6922e183) Enabled non-edge zones available for scan: 1 2025-06-29 04:30:58,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-092a5612]) (logid:6922e183) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:30:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-092a5612]) (logid:6922e183) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:30:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a97145d3]) (logid:ded68e9c) Enabled non-edge zones available for scan: 1 2025-06-29 04:30:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a97145d3]) (logid:ded68e9c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:30:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a18d29db]) (logid:5ef69117) No inactive management server node found 2025-06-29 04:30:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a18d29db]) (logid:5ef69117) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:30:58,496 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3eef4209]) (logid:be6c93db) HA health check task is running... 2025-06-29 04:30:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2560e76c]) (logid:51989ac3) No inactive management server node found 2025-06-29 04:30:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2560e76c]) (logid:51989ac3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57a808ad]) (logid:00858f39) No inactive management server node found 2025-06-29 04:31:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57a808ad]) (logid:00858f39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:02,497 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ae3d2ec8]) (logid:86a246b8) HA health check task is running... 2025-06-29 04:31:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51841e74]) (logid:6f6d8e58) No inactive management server node found 2025-06-29 04:31:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51841e74]) (logid:6f6d8e58) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68f81241]) (logid:12f8f890) No inactive management server node found 2025-06-29 04:31:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68f81241]) (logid:12f8f890) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:05,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8645: Seq 7-8645: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:31:05,371 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8645 2025-06-29 04:31:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59af3508]) (logid:e24bc76c) No inactive management server node found 2025-06-29 04:31:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59af3508]) (logid:e24bc76c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:06,499 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cc9c4860]) (logid:09bfbba8) HA health check task is running... 2025-06-29 04:31:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33d06b01]) (logid:7fd56e64) No inactive management server node found 2025-06-29 04:31:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33d06b01]) (logid:7fd56e64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f526dd01]) (logid:2a086d6c) No inactive management server node found 2025-06-29 04:31:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f526dd01]) (logid:2a086d6c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9818830]) (logid:8f274c5f) No inactive management server node found 2025-06-29 04:31:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9818830]) (logid:8f274c5f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:10,500 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bd692316]) (logid:e176c255) HA health check task is running... 2025-06-29 04:31:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-891dd9dc]) (logid:7d5e72c5) No inactive management server node found 2025-06-29 04:31:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-891dd9dc]) (logid:7d5e72c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ed4d4372]) (logid:edc73612) Found 0 routers to update status. 2025-06-29 04:31:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ed4d4372]) (logid:edc73612) Found 0 VPC's to update Redundant State. 2025-06-29 04:31:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ed4d4372]) (logid:edc73612) Found 0 networks to update RvR status. 2025-06-29 04:31:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c7a8aaf]) (logid:f97b7de2) No inactive management server node found 2025-06-29 04:31:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c7a8aaf]) (logid:f97b7de2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:14,501 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6021656f]) (logid:a1351507) HA health check task is running... 2025-06-29 04:31:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c35be2e]) (logid:d5c6be4a) No inactive management server node found 2025-06-29 04:31:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c35be2e]) (logid:d5c6be4a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:15,351 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8646: Seq 7-8646: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:31:15,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8646 2025-06-29 04:31:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6f65549]) (logid:6d6499ef) No inactive management server node found 2025-06-29 04:31:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6f65549]) (logid:6d6499ef) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f5c8ecd]) (logid:f0ff2cba) No inactive management server node found 2025-06-29 04:31:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f5c8ecd]) (logid:f0ff2cba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:18,043 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-99a676d7]) (logid:4d56ab7f) AutoScaling Monitor is running... 2025-06-29 04:31:18,112 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-f8e8a29b]) (logid:6dc70315) Starting to remove all volume_stats rows older than [Sat Jun 28 16:31:18 CEST 2025]. 2025-06-29 04:31:18,112 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-f8e8a29b]) (logid:6dc70315) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:31:18 CEST 2025]. 2025-06-29 04:31:18,503 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-67d2831e]) (logid:33fb0da4) HA health check task is running... 2025-06-29 04:31:18,744 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-177283cb]) (logid:9f774066) Starting to remove all vm_stats rows older than [Sat Jun 28 16:31:18 CEST 2025]. 2025-06-29 04:31:18,768 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-177283cb]) (logid:9f774066) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:31:18 CEST 2025]. 2025-06-29 04:31:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e2d0105]) (logid:32df7ad9) No inactive management server node found 2025-06-29 04:31:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e2d0105]) (logid:32df7ad9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:20,424 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:31:20,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:31:20,426 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) Seq 2-4725683383995203815: Routed from 90520738888109 2025-06-29 04:31:20,469 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) Seq 1-4725683383995203815: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:31:20,471 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:31:20,472 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) Seq 2-3926857400090362108: Routed from 90520738888109 2025-06-29 04:31:20,517 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-04178b32]) (logid:207dbc46) Seq 2-3926857400090362108: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:31:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61700311]) (logid:5dffafc7) No inactive management server node found 2025-06-29 04:31:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61700311]) (logid:5dffafc7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9756621]) (logid:fb1cf936) No inactive management server node found 2025-06-29 04:31:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9756621]) (logid:fb1cf936) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:22,504 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cd22f475]) (logid:6836ca62) HA health check task is running... 2025-06-29 04:31:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c1a029b]) (logid:caeed380) No inactive management server node found 2025-06-29 04:31:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4c1a029b]) (logid:caeed380) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:24,510 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:31:24,515 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:31:24,516 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) Seq 2-4725683383995203816: Routed from 90520738888109 2025-06-29 04:31:24,569 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) Seq 1-4725683383995203816: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:31:24,603 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:31:24,604 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) Seq 2-3926857400090362109: Routed from 90520738888109 2025-06-29 04:31:24,700 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-f033cb5a]) (logid:aea25086) Seq 2-3926857400090362109: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:31:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d26159]) (logid:c860b974) No inactive management server node found 2025-06-29 04:31:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d26159]) (logid:c860b974) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:25,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8647: Seq 7-8647: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:31:25,368 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8647 2025-06-29 04:31:25,650 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) StorageCollector is running... 2025-06-29 04:31:25,656 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:31:25,656 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:31:25,656 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:31:25,657 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 2-3938116399158788158: Routed from 90520738888109 2025-06-29 04:31:25,713 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 4-3938116399158788158: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:31:25,717 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:31:25,717 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:31:25,717 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:31:25,718 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 2-4725683383995203817: Routed from 90520738888109 2025-06-29 04:31:25,765 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 1-4725683383995203817: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:31:25,776 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:31:25,776 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:31:25,776 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:31:25,777 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 2-4725683383995203818: Routed from 90520738888109 2025-06-29 04:31:25,865 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 1-4725683383995203818: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:31:25,876 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:31:25,876 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:31:25,876 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:31:25,877 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 2-4725683383995203819: Routed from 90520738888109 2025-06-29 04:31:25,962 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 1-4725683383995203819: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:31:25,963 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:31:25,963 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:31:25,963 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:31:25,964 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 2-3926857400090362110: Routed from 90520738888109 2025-06-29 04:31:25,968 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-68d5da3c]) (logid:e98a874b) Seq 2-3926857400090362110: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:31:26,505 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8af48144]) (logid:e774fa73) HA health check task is running... 2025-06-29 04:31:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56fbb14a]) (logid:ca6facdf) No inactive management server node found 2025-06-29 04:31:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56fbb14a]) (logid:ca6facdf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-81447437]) (logid:def7cdf1) Enabled non-edge zones available for scan: 1 2025-06-29 04:31:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-81447437]) (logid:def7cdf1) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:31:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-81447437]) (logid:def7cdf1) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:31:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-358e0366]) (logid:9f378907) Enabled non-edge zones available for scan: 1 2025-06-29 04:31:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-358e0366]) (logid:9f378907) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:31:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7127bf6a]) (logid:bc8a5a38) No inactive management server node found 2025-06-29 04:31:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7127bf6a]) (logid:bc8a5a38) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33798a5d]) (logid:a2ebd405) No inactive management server node found 2025-06-29 04:31:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33798a5d]) (logid:a2ebd405) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:30,507 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-e458aff4]) (logid:02e3d8fd) HA health check task is running... 2025-06-29 04:31:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33868009]) (logid:6046e91a) No inactive management server node found 2025-06-29 04:31:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-33868009]) (logid:6046e91a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:32,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a29e5dc]) (logid:3ae2baf8) No inactive management server node found 2025-06-29 04:31:32,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a29e5dc]) (logid:3ae2baf8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ca21ab9]) (logid:1bc44c78) No inactive management server node found 2025-06-29 04:31:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ca21ab9]) (logid:1bc44c78) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:34,508 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b408bda8]) (logid:09ca826a) HA health check task is running... 2025-06-29 04:31:35,385 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8648: Seq 7-8648: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:31:35,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8648 2025-06-29 04:31:35,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faa01cb1]) (logid:f8f5b70e) No inactive management server node found 2025-06-29 04:31:35,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-faa01cb1]) (logid:f8f5b70e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9171c50]) (logid:2f0e4c6f) No inactive management server node found 2025-06-29 04:31:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9171c50]) (logid:2f0e4c6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:38,509 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-113d5a81]) (logid:c05f68aa) HA health check task is running... 2025-06-29 04:31:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58ee693a]) (logid:db6325ad) No inactive management server node found 2025-06-29 04:31:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-58ee693a]) (logid:db6325ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:40,352 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8649: Seq 7-8649: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:31:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-196e78ce]) (logid:4b2d5e6d) No inactive management server node found 2025-06-29 04:31:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-196e78ce]) (logid:4b2d5e6d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:40,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8649 2025-06-29 04:31:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45dae5bd]) (logid:c5b1ff85) No inactive management server node found 2025-06-29 04:31:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45dae5bd]) (logid:c5b1ff85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:42,512 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6fd57f8e]) (logid:904b91d2) HA health check task is running... 2025-06-29 04:31:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-22ec9a2d]) (logid:03bea856) DbCollector is running... 2025-06-29 04:31:43,095 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-e3374d93]) (logid:2fd062b1) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:31:43,096 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-e3374d93]) (logid:2fd062b1) Snapshot scheduler is being called at [2025-06-29T02:31:43+0000]. 2025-06-29 04:31:43,097 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-e3374d93]) (logid:2fd062b1) There are [0] scheduled snapshots to be executed at [2025-06-29T02:31:43+0000]. 2025-06-29 04:31:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-a21cff71]) (logid:987559dd) Found 0 running routers. 2025-06-29 04:31:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d1756a61]) (logid:d9813449) Found 0 routers to update status. 2025-06-29 04:31:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d1756a61]) (logid:d9813449) Found 0 VPC's to update Redundant State. 2025-06-29 04:31:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d1756a61]) (logid:d9813449) Found 0 networks to update RvR status. 2025-06-29 04:31:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-e9953269]) (logid:b576ebd9) External devices are not used. Skipping external device usage collection 2025-06-29 04:31:43,134 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-30cd85f7]) (logid:4c7cb8a3) Started resource counters recalculation periodic task. 2025-06-29 04:31:43,136 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-30cd85f7]) (logid:4c7cb8a3) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:31:43,305 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-56450424]) (logid:de04b4a8) Backup backup.poll is being called at 2025-06-29T02:31:43+0000 2025-06-29 04:31:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5570b75]) (logid:6ed6c760) No inactive management server node found 2025-06-29 04:31:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5570b75]) (logid:6ed6c760) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:43,513 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-30cd85f7]) (logid:4c7cb8a3) Finished resource counters recalculation periodic task. 2025-06-29 04:31:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) ManagementServerCollector is running... 2025-06-29 04:31:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) DB properties were already loaded 2025-06-29 04:31:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:31:44,059 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Execution of process [4067201] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:31:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Exit value of process [4067201] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:31:44,061 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Process [4067201] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:31:44,061 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) usage install: null 2025-06-29 04:31:44,061 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Sessions active: 0 2025-06-29 04:31:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:31:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067204] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:31:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:31:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067209] for command [/bin/bash -c uname -r ]. 2025-06-29 04:31:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:31:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067210] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:31:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) system memory from /proc: 4887515136 2025-06-29 04:31:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:31:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067215] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:31:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) free memory from /proc: 340750336 2025-06-29 04:31:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:31:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067220] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:31:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) used memory from /proc: 1024616 2025-06-29 04:31:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:31:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067221] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:31:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:31:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067222] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:31:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:31:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067223] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:31:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:31:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067230] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:31:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:31:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Successfully executed process [4067231] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:31:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:31:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail buffersdirect.count, 93 2025-06-29 04:31:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail buffersdirect.used, 321668 2025-06-29 04:31:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail buffersmapped.capacity, 0 2025-06-29 04:31:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail buffersmapped.count, 0 2025-06-29 04:31:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail buffersmapped.used, 0 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail jvmuptime, 3363834 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memoryheap.committed, 638058496 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memoryheap.init, 77594624 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memoryheap.usage, 0.19548093758582807 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorynon-heap.committed, 198311936 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorynon-heap.max, -1 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorynon-heap.usage, -1.95879232E8 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorynon-heap.used, 195879232 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11735724640095975 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22769062218815608 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.Metaspace.usage, 0.9928163888031852 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.PS-Eden-Space.usage, 0.28511443187753327 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorytotal.committed, 836370432 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorytotal.init, 85262336 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorytotal.max, 1908932607 2025-06-29 04:31:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail memorytotal.used, 569039168 2025-06-29 04:31:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail threadsdeadlocks, [] 2025-06-29 04:31:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail threadsnew.count, 0 2025-06-29 04:31:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:31:44,134 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-af6dd7c9]) (logid:67ebbcd5) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:31:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":265582216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3363764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":568355512,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":373159936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":340750336,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.2],"systemCyclesUsage":[66416951,53837063,4589899389],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:31:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 111, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:31:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":265582216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3363764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":568355512,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":373159936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":340750336,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.2],"systemCyclesUsage":[66416951,53837063,4589899389],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:31:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:31:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-24:[ctx-682978de]) (logid:f35c07ce) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:31:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":265582216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3363764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":568355512,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":373159936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":340750336,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.2],"systemCyclesUsage":[66416951,53837063,4589899389],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:31:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:31:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 111, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:31:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":265582216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3363764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":568355512,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":373159936,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":340750336,"systemMemoryUsed":1024616,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.2],"systemCyclesUsage":[66416951,53837063,4589899389],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:31:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c8733f0]) (logid:bcd27105) No inactive management server node found 2025-06-29 04:31:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c8733f0]) (logid:bcd27105) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:31:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:31:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e6999a6]) (logid:7001a882) No inactive management server node found 2025-06-29 04:31:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e6999a6]) (logid:7001a882) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:31:46,513 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d0ff0a48]) (logid:eaaab7a5) HA health check task is running... 2025-06-29 04:31:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:31:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:31:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:31:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:31:47,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:31:47,118 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:31:47,119 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:31:47,121 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:31:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe59a5a3]) (logid:6ea10304) No inactive management server node found 2025-06-29 04:31:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe59a5a3]) (logid:6ea10304) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:48,054 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-6:[ctx-6b4e710d]) (logid:967daff2) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:31:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ee535b87]) (logid:d0beee7f) VM scheduler.poll is being called at 2025-06-29T02:32:00+0000 2025-06-29 04:31:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ee535b87]) (logid:d0beee7f) Got 0 scheduled jobs to be executed at 2025-06-29T02:32:00+0000 2025-06-29 04:31:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-ee535b87]) (logid:d0beee7f) Cleaned up 0 VM scheduled job entries 2025-06-29 04:31:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-4c4ca23f]) (logid:51f1150c) VM Operation Thread Running 2025-06-29 04:31:48,350 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6bb3e407]) (logid:d7bc4583) ClusterDRS.poll is being called at 2025-06-29T02:32:00+0000 2025-06-29 04:31:48,357 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6bb3e407]) (logid:d7bc4583) Removed 0 old drs migration plans 2025-06-29 04:31:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7b514fc]) (logid:cb37c11c) No inactive management server node found 2025-06-29 04:31:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7b514fc]) (logid:cb37c11c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:50,360 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8651: Seq 7-8651: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:31:50,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8651 2025-06-29 04:31:50,514 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6120690b]) (logid:fbe32752) HA health check task is running... 2025-06-29 04:31:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d7ef338]) (logid:18a6ebba) No inactive management server node found 2025-06-29 04:31:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d7ef338]) (logid:18a6ebba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cddada6]) (logid:cd680e17) No inactive management server node found 2025-06-29 04:31:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3cddada6]) (logid:cd680e17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c691d4af]) (logid:cbb38ae8) No inactive management server node found 2025-06-29 04:31:53,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c691d4af]) (logid:cbb38ae8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:54,515 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-50106d27]) (logid:196c676f) HA health check task is running... 2025-06-29 04:31:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ea1d3cc]) (logid:4a593742) No inactive management server node found 2025-06-29 04:31:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ea1d3cc]) (logid:4a593742) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:56,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9342b8ab]) (logid:1140a4d0) No inactive management server node found 2025-06-29 04:31:56,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9342b8ab]) (logid:1140a4d0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f288efce]) (logid:247e2bd9) Enabled non-edge zones available for scan: 1 2025-06-29 04:31:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f288efce]) (logid:247e2bd9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:31:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f288efce]) (logid:247e2bd9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:31:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-055665e7]) (logid:d4105fc0) Enabled non-edge zones available for scan: 1 2025-06-29 04:31:58,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-055665e7]) (logid:d4105fc0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:31:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91d6853b]) (logid:37671768) No inactive management server node found 2025-06-29 04:31:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91d6853b]) (logid:37671768) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:31:58,517 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-44350b30]) (logid:eea27b5f) HA health check task is running... 2025-06-29 04:31:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-810df1c0]) (logid:800e6192) No inactive management server node found 2025-06-29 04:31:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-810df1c0]) (logid:800e6192) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:00,353 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8652: Seq 7-8652: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:00,363 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8652 2025-06-29 04:32:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76c657e0]) (logid:0c521c56) No inactive management server node found 2025-06-29 04:32:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76c657e0]) (logid:0c521c56) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:02,518 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-15c70619]) (logid:48e78edf) HA health check task is running... 2025-06-29 04:32:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b97f2bd]) (logid:880442de) No inactive management server node found 2025-06-29 04:32:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b97f2bd]) (logid:880442de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e470910]) (logid:2d4335f8) No inactive management server node found 2025-06-29 04:32:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e470910]) (logid:2d4335f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab22a3d5]) (logid:a3637c99) No inactive management server node found 2025-06-29 04:32:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab22a3d5]) (logid:a3637c99) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:06,519 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-18710352]) (logid:75e1bb8f) HA health check task is running... 2025-06-29 04:32:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbd67d1c]) (logid:6185d4b0) No inactive management server node found 2025-06-29 04:32:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cbd67d1c]) (logid:6185d4b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a625c9c]) (logid:f7693b3a) No inactive management server node found 2025-06-29 04:32:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a625c9c]) (logid:f7693b3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:10,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8653: Seq 7-8653: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-609404c7]) (logid:fac52ccf) No inactive management server node found 2025-06-29 04:32:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-609404c7]) (logid:fac52ccf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:10,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8653 2025-06-29 04:32:10,521 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3d8ef120]) (logid:886603e4) HA health check task is running... 2025-06-29 04:32:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44e0fa88]) (logid:a49c53bb) No inactive management server node found 2025-06-29 04:32:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44e0fa88]) (logid:a49c53bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b6e8111d]) (logid:27f8f5c1) Checking if any host reservation can be released ... 2025-06-29 04:32:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b6e8111d]) (logid:27f8f5c1) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:13,016 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b6e8111d]) (logid:27f8f5c1) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:13,016 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-b6e8111d]) (logid:27f8f5c1) Done running HostReservationReleaseChecker ... 2025-06-29 04:32:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c784cc57]) (logid:5bb14262) Found 0 routers to update status. 2025-06-29 04:32:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c784cc57]) (logid:5bb14262) Found 0 VPC's to update Redundant State. 2025-06-29 04:32:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c784cc57]) (logid:5bb14262) Found 0 networks to update RvR status. 2025-06-29 04:32:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Running Capacity Checker ... 2025-06-29 04:32:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) recalculating system capacity 2025-06-29 04:32:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Executing cpu/ram capacity update 2025-06-29 04:32:13,169 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:13,182 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:13,189 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:32:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:32:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:32:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:13,205 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:13,214 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:32:13,214 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:32:13,215 DEBUG [c.c.c.CapacityManagerImpl] (pool-34-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:32:13,225 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Done executing cpu/ram capacity update 2025-06-29 04:32:13,225 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Executing storage capacity update 2025-06-29 04:32:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:32:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:32:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:32:13,240 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:32:13,245 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:32:13,246 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:32:13,246 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:32:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:32:13,254 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:32:13,254 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:32:13,254 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:32:13,259 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:32:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:32:13,265 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:32:13,265 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:32:13,268 DEBUG [c.c.s.StorageManagerImpl] (pool-35-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:32:13,269 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Done executing storage capacity update 2025-06-29 04:32:13,269 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Executing capacity updates for public ip and Vlans 2025-06-29 04:32:13,289 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Done capacity updates for public ip and Vlans 2025-06-29 04:32:13,290 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Executing capacity updates for private ip 2025-06-29 04:32:13,296 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Done executing capacity updates for private ip 2025-06-29 04:32:13,296 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Done recalculating system capacity 2025-06-29 04:32:13,345 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-bb53cb74]) (logid:44d058e3) Done running Capacity Checker ... 2025-06-29 04:32:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04c960cd]) (logid:3f23157a) No inactive management server node found 2025-06-29 04:32:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04c960cd]) (logid:3f23157a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:14,522 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4a457e3e]) (logid:caeb2f40) HA health check task is running... 2025-06-29 04:32:14,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0811499]) (logid:08b3ceba) No inactive management server node found 2025-06-29 04:32:14,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0811499]) (logid:08b3ceba) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-876966d9]) (logid:472829ac) No inactive management server node found 2025-06-29 04:32:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-876966d9]) (logid:472829ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:17,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ffe0f3f]) (logid:fa30dbd9) No inactive management server node found 2025-06-29 04:32:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ffe0f3f]) (logid:fa30dbd9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:18,045 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-417fbf74]) (logid:82233a67) AutoScaling Monitor is running... 2025-06-29 04:32:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-06c9954f]) (logid:37806ef7) Resetting hosts suitable for reconnect 2025-06-29 04:32:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-06c9954f]) (logid:37806ef7) Completed resetting hosts suitable for reconnect 2025-06-29 04:32:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-06c9954f]) (logid:37806ef7) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:32:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-06c9954f]) (logid:37806ef7) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:32:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-06c9954f]) (logid:37806ef7) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:32:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-06c9954f]) (logid:37806ef7) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:32:18,115 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b136d7b7]) (logid:d2775a3d) Starting to remove all volume_stats rows older than [Sat Jun 28 16:32:18 CEST 2025]. 2025-06-29 04:32:18,115 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b136d7b7]) (logid:d2775a3d) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:32:18 CEST 2025]. 2025-06-29 04:32:18,524 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4cb0f84d]) (logid:f8455d6c) HA health check task is running... 2025-06-29 04:32:18,770 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-3123c53a]) (logid:b8f01e51) Starting to remove all vm_stats rows older than [Sat Jun 28 16:32:18 CEST 2025]. 2025-06-29 04:32:18,794 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-3123c53a]) (logid:b8f01e51) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:32:18 CEST 2025]. 2025-06-29 04:32:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e0d58b4]) (logid:d588dc05) No inactive management server node found 2025-06-29 04:32:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e0d58b4]) (logid:d588dc05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:20,354 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8654: Seq 7-8654: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:20,363 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8654 2025-06-29 04:32:20,520 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:32:20,523 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:32:20,525 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) Seq 2-4725683383995203820: Routed from 90520738888109 2025-06-29 04:32:20,567 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) Seq 1-4725683383995203820: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:32:20,569 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:32:20,570 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) Seq 2-3926857400090362111: Routed from 90520738888109 2025-06-29 04:32:20,614 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-b1a7e275]) (logid:5239235a) Seq 2-3926857400090362111: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:32:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e691c589]) (logid:d46f3cd4) No inactive management server node found 2025-06-29 04:32:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e691c589]) (logid:d46f3cd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc2ebc98]) (logid:57546cb5) No inactive management server node found 2025-06-29 04:32:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc2ebc98]) (logid:57546cb5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:22,525 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-bb913423]) (logid:79bedffc) HA health check task is running... 2025-06-29 04:32:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5638f562]) (logid:0c145fd2) No inactive management server node found 2025-06-29 04:32:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5638f562]) (logid:0c145fd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:24,760 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:32:24,765 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:32:24,766 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) Seq 2-4725683383995203821: Routed from 90520738888109 2025-06-29 04:32:24,820 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) Seq 1-4725683383995203821: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:32:24,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:32:24,859 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) Seq 2-3926857400090362112: Routed from 90520738888109 2025-06-29 04:32:24,952 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-64b93b10]) (logid:5053e174) Seq 2-3926857400090362112: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:32:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54d87c1e]) (logid:25494d10) No inactive management server node found 2025-06-29 04:32:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54d87c1e]) (logid:25494d10) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:25,968 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) StorageCollector is running... 2025-06-29 04:32:25,976 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:32:25,976 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:32:25,976 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:32:25,977 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-3938116399158788159: Routed from 90520738888109 2025-06-29 04:32:26,032 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 4-3938116399158788159: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:32:26,038 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:32:26,038 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:32:26,038 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:32:26,039 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-4725683383995203822: Routed from 90520738888109 2025-06-29 04:32:26,087 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 1-4725683383995203822: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:32:26,099 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:32:26,099 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:32:26,099 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:32:26,100 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-3926857400090362113: Routed from 90520738888109 2025-06-29 04:32:26,114 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-3926857400090362113: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:32:26,124 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:32:26,124 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:32:26,124 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:32:26,125 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-4725683383995203823: Routed from 90520738888109 2025-06-29 04:32:26,209 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 1-4725683383995203823: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:32:26,211 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:32:26,211 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:32:26,211 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:32:26,213 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-3926857400090362114: Routed from 90520738888109 2025-06-29 04:32:26,257 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-5c72e5ff]) (logid:419ff1d6) Seq 2-3926857400090362114: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:32:26,526 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-580eaf9d]) (logid:0f3c0262) HA health check task is running... 2025-06-29 04:32:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52ceb56b]) (logid:a4b2ba05) No inactive management server node found 2025-06-29 04:32:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52ceb56b]) (logid:a4b2ba05) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-995327e5]) (logid:afa9d87b) Enabled non-edge zones available for scan: 1 2025-06-29 04:32:28,097 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-995327e5]) (logid:afa9d87b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:32:28,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-995327e5]) (logid:afa9d87b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:32:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4f62f37d]) (logid:eddb2753) Enabled non-edge zones available for scan: 1 2025-06-29 04:32:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4f62f37d]) (logid:eddb2753) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:32:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f067d8eb]) (logid:ccc4a035) No inactive management server node found 2025-06-29 04:32:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f067d8eb]) (logid:ccc4a035) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d885182]) (logid:12f9c2a5) No inactive management server node found 2025-06-29 04:32:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d885182]) (logid:12f9c2a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:30,355 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8655: Seq 7-8655: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:30,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8655 2025-06-29 04:32:30,527 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0d73622c]) (logid:6fc5e022) HA health check task is running... 2025-06-29 04:32:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03bc3c3d]) (logid:b453ae05) No inactive management server node found 2025-06-29 04:32:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03bc3c3d]) (logid:b453ae05) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d07042b]) (logid:6c70b838) No inactive management server node found 2025-06-29 04:32:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d07042b]) (logid:6c70b838) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0a7604b]) (logid:0f288a40) No inactive management server node found 2025-06-29 04:32:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0a7604b]) (logid:0f288a40) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:34,529 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b7760b84]) (logid:abedc9b7) HA health check task is running... 2025-06-29 04:32:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f91374e7]) (logid:db051b6d) No inactive management server node found 2025-06-29 04:32:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f91374e7]) (logid:db051b6d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67f3251c]) (logid:c834bf28) No inactive management server node found 2025-06-29 04:32:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67f3251c]) (logid:c834bf28) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:38,530 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ee05c447]) (logid:a07ce25d) HA health check task is running... 2025-06-29 04:32:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f387d24]) (logid:186d1b4b) No inactive management server node found 2025-06-29 04:32:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f387d24]) (logid:186d1b4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:40,355 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8656: Seq 7-8656: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5139a43b]) (logid:327345fc) No inactive management server node found 2025-06-29 04:32:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5139a43b]) (logid:327345fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:40,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8656 2025-06-29 04:32:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3540ccb7]) (logid:84a35572) No inactive management server node found 2025-06-29 04:32:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3540ccb7]) (logid:84a35572) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:42,532 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e85a01a2]) (logid:37ed2de4) HA health check task is running... 2025-06-29 04:32:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-dca77aff]) (logid:f66a38f7) DbCollector is running... 2025-06-29 04:32:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-48392867]) (logid:09a7674c) Found 0 routers to update status. 2025-06-29 04:32:43,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-48392867]) (logid:09a7674c) Found 0 VPC's to update Redundant State. 2025-06-29 04:32:43,103 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-48392867]) (logid:09a7674c) Found 0 networks to update RvR status. 2025-06-29 04:32:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e15947bc]) (logid:4ad0856e) No inactive management server node found 2025-06-29 04:32:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e15947bc]) (logid:4ad0856e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) ManagementServerCollector is running... 2025-06-29 04:32:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) DB properties were already loaded 2025-06-29 04:32:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:32:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Execution of process [4067247] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:32:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Exit value of process [4067247] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:32:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Process [4067247] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:32:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) usage install: null 2025-06-29 04:32:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Sessions active: 0 2025-06-29 04:32:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:32:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067250] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:32:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:32:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067255] for command [/bin/bash -c uname -r ]. 2025-06-29 04:32:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:32:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067256] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:32:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) system memory from /proc: 4887515136 2025-06-29 04:32:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:32:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067261] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:32:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) free memory from /proc: 346619904 2025-06-29 04:32:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:32:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067266] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:32:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) used memory from /proc: 1024744 2025-06-29 04:32:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:32:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067267] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:32:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:32:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067268] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:32:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:32:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067269] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:32:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:32:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067276] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:32:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:32:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Successfully executed process [4067277] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:32:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:32:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail buffersdirect.count, 93 2025-06-29 04:32:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail buffersdirect.used, 321668 2025-06-29 04:32:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail buffersmapped.capacity, 0 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail buffersmapped.count, 0 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail buffersmapped.used, 0 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail jvmuptime, 3423831 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memoryheap.committed, 638058496 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memoryheap.init, 77594624 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memoryheap.usage, 0.2113995550753356 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorynon-heap.committed, 198311936 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorynon-heap.max, -1 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorynon-heap.usage, -1.93139824E8 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorynon-heap.used, 193139824 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1123991935483871 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21029051554637251 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.Metaspace.usage, 0.9928782561265868 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.PS-Eden-Space.usage, 0.33187740377849156 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorytotal.committed, 836370432 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorytotal.init, 85262336 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorytotal.max, 1908932607 2025-06-29 04:32:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail memorytotal.used, 596921824 2025-06-29 04:32:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail threadsdeadlocks, [] 2025-06-29 04:32:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail threadsnew.count, 0 2025-06-29 04:32:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:32:44,132 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-3b2738f6]) (logid:b3e5cf1c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:32:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":235280304,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3423760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":595916736,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":403547504,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346619904,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.03,0.18],"systemCyclesUsage":[66417374,53837388,4589922379],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:32:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 113, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:32:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":235280304,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3423760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":595916736,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":403547504,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346619904,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.03,0.18],"systemCyclesUsage":[66417374,53837388,4589922379],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:32:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:32:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-24:[ctx-5d82615f]) (logid:bf1357c1) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:32:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":235280304,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3423760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":595916736,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":403547504,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346619904,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.03,0.18],"systemCyclesUsage":[66417374,53837388,4589922379],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:32:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 8 ms 2025-06-29 04:32:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 113, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:32:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":235280304,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3423760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":595916736,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":403547504,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":346619904,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.03,0.18],"systemCyclesUsage":[66417374,53837388,4589922379],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:32:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e89538a2]) (logid:862a6153) No inactive management server node found 2025-06-29 04:32:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e89538a2]) (logid:862a6153) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:45,355 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8657: Seq 7-8657: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:45,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8657 2025-06-29 04:32:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:32:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:32:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7266ac3]) (logid:4a274d51) No inactive management server node found 2025-06-29 04:32:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7266ac3]) (logid:4a274d51) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:46,533 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ce171ec5]) (logid:c4a38c31) HA health check task is running... 2025-06-29 04:32:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:32:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:32:47,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:47,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:32:47,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:32:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16645384]) (logid:14a81a9f) No inactive management server node found 2025-06-29 04:32:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16645384]) (logid:14a81a9f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f646878f]) (logid:a3f95d09) VM scheduler.poll is being called at 2025-06-29T02:33:00+0000 2025-06-29 04:32:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f646878f]) (logid:a3f95d09) Got 0 scheduled jobs to be executed at 2025-06-29T02:33:00+0000 2025-06-29 04:32:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f646878f]) (logid:a3f95d09) Cleaned up 0 VM scheduled job entries 2025-06-29 04:32:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-937a3ad1]) (logid:ab368ab3) VM Operation Thread Running 2025-06-29 04:32:48,350 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1147a58f]) (logid:13c3e171) ClusterDRS.poll is being called at 2025-06-29T02:33:00+0000 2025-06-29 04:32:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1147a58f]) (logid:13c3e171) Removed 0 old drs migration plans 2025-06-29 04:32:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57a17354]) (logid:0ff69cd5) No inactive management server node found 2025-06-29 04:32:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57a17354]) (logid:0ff69cd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:50,534 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0854db9d]) (logid:75a0498e) HA health check task is running... 2025-06-29 04:32:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f708a8]) (logid:db2a83f8) No inactive management server node found 2025-06-29 04:32:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f708a8]) (logid:db2a83f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:52,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32249241]) (logid:f6525a99) No inactive management server node found 2025-06-29 04:32:52,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32249241]) (logid:f6525a99) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34b8ec25]) (logid:1218a6c0) No inactive management server node found 2025-06-29 04:32:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34b8ec25]) (logid:1218a6c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:54,536 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0111fac1]) (logid:3fd05e49) HA health check task is running... 2025-06-29 04:32:55,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8659: Seq 7-8659: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:32:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bf1eb68]) (logid:0ae03b71) No inactive management server node found 2025-06-29 04:32:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6bf1eb68]) (logid:0ae03b71) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:55,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8659 2025-06-29 04:32:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe253e6b]) (logid:1b5f9ece) No inactive management server node found 2025-06-29 04:32:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe253e6b]) (logid:1b5f9ece) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2b4af54f]) (logid:be250853) Enabled non-edge zones available for scan: 1 2025-06-29 04:32:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-2b4af54f]) (logid:be250853) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:32:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2b4af54f]) (logid:be250853) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:32:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9b4f57b0]) (logid:79fce2d8) Enabled non-edge zones available for scan: 1 2025-06-29 04:32:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9b4f57b0]) (logid:79fce2d8) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:32:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-372e5ac0]) (logid:591f59d7) No inactive management server node found 2025-06-29 04:32:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-372e5ac0]) (logid:591f59d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:32:58,537 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8d1e4fdf]) (logid:74a92aaf) HA health check task is running... 2025-06-29 04:32:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283cfcf2]) (logid:fa2b4748) No inactive management server node found 2025-06-29 04:32:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-283cfcf2]) (logid:fa2b4748) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a27b263]) (logid:c189780d) No inactive management server node found 2025-06-29 04:33:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a27b263]) (logid:c189780d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:02,538 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cb85fabe]) (logid:d48ef9e4) HA health check task is running... 2025-06-29 04:33:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ae198e2]) (logid:8798ba47) No inactive management server node found 2025-06-29 04:33:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ae198e2]) (logid:8798ba47) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b2ea7c]) (logid:4b5e638f) No inactive management server node found 2025-06-29 04:33:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7b2ea7c]) (logid:4b5e638f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:05,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8660: Seq 7-8660: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:33:05,366 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8660 2025-06-29 04:33:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efe51261]) (logid:b32c6c8c) No inactive management server node found 2025-06-29 04:33:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efe51261]) (logid:b32c6c8c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:06,540 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4cdf34e7]) (logid:2db696e0) HA health check task is running... 2025-06-29 04:33:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac71b51f]) (logid:c9f4214d) No inactive management server node found 2025-06-29 04:33:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac71b51f]) (logid:c9f4214d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e26a539]) (logid:5e688750) No inactive management server node found 2025-06-29 04:33:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e26a539]) (logid:5e688750) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ac121eb]) (logid:1bef595b) No inactive management server node found 2025-06-29 04:33:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2ac121eb]) (logid:1bef595b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:10,541 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-05eaa409]) (logid:00827871) HA health check task is running... 2025-06-29 04:33:11,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1926167]) (logid:8ab5936b) No inactive management server node found 2025-06-29 04:33:11,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b1926167]) (logid:8ab5936b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2f90e9da]) (logid:d9667e79) Found 0 routers to update status. 2025-06-29 04:33:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2f90e9da]) (logid:d9667e79) Found 0 VPC's to update Redundant State. 2025-06-29 04:33:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2f90e9da]) (logid:d9667e79) Found 0 networks to update RvR status. 2025-06-29 04:33:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed41d06b]) (logid:222950f5) No inactive management server node found 2025-06-29 04:33:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed41d06b]) (logid:222950f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:14,542 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d7ede32f]) (logid:eb454412) HA health check task is running... 2025-06-29 04:33:14,863 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ed98d9c]) (logid:699348d3) No inactive management server node found 2025-06-29 04:33:14,863 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ed98d9c]) (logid:699348d3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:15,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8661: Seq 7-8661: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:33:15,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8661 2025-06-29 04:33:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95c4c01d]) (logid:8e205e69) No inactive management server node found 2025-06-29 04:33:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95c4c01d]) (logid:8e205e69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:17,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26758152]) (logid:0411a478) No inactive management server node found 2025-06-29 04:33:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26758152]) (logid:0411a478) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:18,047 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-d54e5915]) (logid:22df976a) AutoScaling Monitor is running... 2025-06-29 04:33:18,118 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-b50125ba]) (logid:548c27ee) Starting to remove all volume_stats rows older than [Sat Jun 28 16:33:18 CEST 2025]. 2025-06-29 04:33:18,119 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-b50125ba]) (logid:548c27ee) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:33:18 CEST 2025]. 2025-06-29 04:33:18,543 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-496c9d5d]) (logid:27ed9c20) HA health check task is running... 2025-06-29 04:33:18,796 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-f8de304b]) (logid:804cefbb) Starting to remove all vm_stats rows older than [Sat Jun 28 16:33:18 CEST 2025]. 2025-06-29 04:33:18,817 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-f8de304b]) (logid:804cefbb) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:33:18 CEST 2025]. 2025-06-29 04:33:19,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12527c2d]) (logid:f82fe30c) No inactive management server node found 2025-06-29 04:33:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12527c2d]) (logid:f82fe30c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:33:20,617 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:33:20,619 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:33:20,621 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) Seq 2-4725683383995203824: Routed from 90520738888109 2025-06-29 04:33:20,664 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) Seq 1-4725683383995203824: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:33:20,666 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:33:20,666 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) Seq 2-3926857400090362115: Routed from 90520738888109 2025-06-29 04:33:20,710 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-8daf708c]) (logid:23bc04ef) Seq 2-3926857400090362115: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:33:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4563b763]) (logid:86e31d8e) No inactive management server node found 2025-06-29 04:33:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4563b763]) (logid:86e31d8e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9efcb87c]) (logid:98103b70) No inactive management server node found 2025-06-29 04:33:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9efcb87c]) (logid:98103b70) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:22,545 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-20cd3ae0]) (logid:8d51fc87) HA health check task is running... 2025-06-29 04:33:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39d93021]) (logid:01be3092) No inactive management server node found 2025-06-29 04:33:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39d93021]) (logid:01be3092) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:25,010 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:33:25,013 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:33:25,014 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) Seq 2-4725683383995203825: Routed from 90520738888109 2025-06-29 04:33:25,069 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) Seq 1-4725683383995203825: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:33:25,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:33:25,103 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) Seq 2-3926857400090362116: Routed from 90520738888109 2025-06-29 04:33:25,218 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-dd397140]) (logid:62ae0761) Seq 2-3926857400090362116: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:33:25,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8662: Seq 7-8662: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:33:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fae77cfe]) (logid:afd447c3) No inactive management server node found 2025-06-29 04:33:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fae77cfe]) (logid:afd447c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:25,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8662 2025-06-29 04:33:26,258 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) StorageCollector is running... 2025-06-29 04:33:26,264 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:33:26,264 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:33:26,264 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:33:26,265 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-3938116399158788160: Routed from 90520738888109 2025-06-29 04:33:26,323 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 4-3938116399158788160: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:33:26,328 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:33:26,328 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:33:26,328 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:33:26,330 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-4725683383995203826: Routed from 90520738888109 2025-06-29 04:33:26,379 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 1-4725683383995203826: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:33:26,390 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:33:26,390 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:33:26,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:33:26,391 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-3926857400090362117: Routed from 90520738888109 2025-06-29 04:33:26,402 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-3926857400090362117: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:33:26,413 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:33:26,413 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:33:26,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:33:26,414 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-4725683383995203827: Routed from 90520738888109 2025-06-29 04:33:26,499 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 1-4725683383995203827: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:33:26,501 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:33:26,502 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:33:26,502 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:33:26,503 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-3926857400090362118: Routed from 90520738888109 2025-06-29 04:33:26,546 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f7ac99a2]) (logid:7dfb4a9f) HA health check task is running... 2025-06-29 04:33:26,547 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e5a5db42]) (logid:f68fc226) Seq 2-3926857400090362118: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:33:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7333ca9]) (logid:e2c24b26) No inactive management server node found 2025-06-29 04:33:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7333ca9]) (logid:e2c24b26) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-912ea4b4]) (logid:2dc3db15) Enabled non-edge zones available for scan: 1 2025-06-29 04:33:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-912ea4b4]) (logid:2dc3db15) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:33:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-912ea4b4]) (logid:2dc3db15) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:33:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e77d421e]) (logid:c1dc3f36) Enabled non-edge zones available for scan: 1 2025-06-29 04:33:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e77d421e]) (logid:c1dc3f36) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:33:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1045c4d7]) (logid:c6d3ce07) No inactive management server node found 2025-06-29 04:33:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1045c4d7]) (logid:c6d3ce07) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e9a6925]) (logid:ecd402b6) No inactive management server node found 2025-06-29 04:33:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e9a6925]) (logid:ecd402b6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:30,358 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8663: Seq 7-8663: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:33:30,368 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8663 2025-06-29 04:33:30,547 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-31dbd912]) (logid:143cf1ef) HA health check task is running... 2025-06-29 04:33:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2aba7bc4]) (logid:171126d1) No inactive management server node found 2025-06-29 04:33:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2aba7bc4]) (logid:171126d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec9640b1]) (logid:30e5ddc2) No inactive management server node found 2025-06-29 04:33:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec9640b1]) (logid:30e5ddc2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cea309d]) (logid:fab6e564) No inactive management server node found 2025-06-29 04:33:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cea309d]) (logid:fab6e564) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:34,548 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f3219db3]) (logid:9759d909) HA health check task is running... 2025-06-29 04:33:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-037fa462]) (logid:1be71e13) No inactive management server node found 2025-06-29 04:33:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-037fa462]) (logid:1be71e13) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e7c3026]) (logid:e927bff4) No inactive management server node found 2025-06-29 04:33:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e7c3026]) (logid:e927bff4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:38,549 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f536a699]) (logid:1ed79cd0) HA health check task is running... 2025-06-29 04:33:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75e9606f]) (logid:5cff107b) No inactive management server node found 2025-06-29 04:33:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75e9606f]) (logid:5cff107b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cf91749]) (logid:2d986cea) No inactive management server node found 2025-06-29 04:33:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0cf91749]) (logid:2d986cea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:40,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8664: Seq 7-8664: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:33:40,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8664 2025-06-29 04:33:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08d4c66c]) (logid:657b0bb2) No inactive management server node found 2025-06-29 04:33:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08d4c66c]) (logid:657b0bb2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:42,551 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3933e30f]) (logid:1a6d6276) HA health check task is running... 2025-06-29 04:33:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-14768569]) (logid:f8905684) DbCollector is running... 2025-06-29 04:33:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-373a42ca]) (logid:82e5c83a) Found 0 routers to update status. 2025-06-29 04:33:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-373a42ca]) (logid:82e5c83a) Found 0 VPC's to update Redundant State. 2025-06-29 04:33:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-373a42ca]) (logid:82e5c83a) Found 0 networks to update RvR status. 2025-06-29 04:33:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cb8502d]) (logid:d8148c06) No inactive management server node found 2025-06-29 04:33:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cb8502d]) (logid:d8148c06) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) ManagementServerCollector is running... 2025-06-29 04:33:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) DB properties were already loaded 2025-06-29 04:33:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:33:44,054 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Execution of process [4067283] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:33:44,054 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Exit value of process [4067283] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:33:44,054 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Process [4067283] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:33:44,054 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) usage install: null 2025-06-29 04:33:44,054 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Sessions active: 0 2025-06-29 04:33:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:33:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067286] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:33:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:33:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067291] for command [/bin/bash -c uname -r ]. 2025-06-29 04:33:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:33:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067292] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:33:44,072 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) system memory from /proc: 4887515136 2025-06-29 04:33:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:33:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067297] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:33:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) free memory from /proc: 333848576 2025-06-29 04:33:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:33:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067302] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:33:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) used memory from /proc: 1024744 2025-06-29 04:33:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:33:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067303] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:33:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:33:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067304] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:33:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:33:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067305] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:33:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:33:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067312] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:33:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:33:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Successfully executed process [4067313] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail buffersdirect.count, 93 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail buffersdirect.used, 321668 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail buffersmapped.capacity, 0 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail buffersmapped.count, 0 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail buffersmapped.used, 0 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail jvmuptime, 3483829 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memoryheap.committed, 638058496 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memoryheap.init, 77594624 2025-06-29 04:33:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memoryheap.usage, 0.22311022935808114 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorynon-heap.committed, 198311936 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorynon-heap.max, -1 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorynon-heap.usage, -1.93226792E8 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorynon-heap.used, 193226792 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11272723107171422 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21063314393308227 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.Metaspace.usage, 0.9929107967317524 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.PS-Eden-Space.usage, 0.36565746738015414 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorytotal.committed, 836370432 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorytotal.init, 85262336 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorytotal.max, 1908932607 2025-06-29 04:33:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail memorytotal.used, 619129184 2025-06-29 04:33:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail threadsdeadlocks, [] 2025-06-29 04:33:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail threadsnew.count, 0 2025-06-29 04:33:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:33:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-8e609b64]) (logid:8d35a69a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:33:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":212854536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3483758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":618428576,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":425902392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":333848576,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.17],"systemCyclesUsage":[66417724,53837700,4589945468],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:33:44,130 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 115, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:33:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":212854536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3483758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":618428576,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":425902392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":333848576,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.17],"systemCyclesUsage":[66417724,53837700,4589945468],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:33:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:33:44,135 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-24:[ctx-7f27b2fc]) (logid:5f0e472d) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:33:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":212854536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3483758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":618428576,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":425902392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":333848576,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.17],"systemCyclesUsage":[66417724,53837700,4589945468],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:33:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:33:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 115, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:33:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":212854536,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3483758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":618428576,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":425902392,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":333848576,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.17],"systemCyclesUsage":[66417724,53837700,4589945468],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:33:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6401d643]) (logid:e329bd2c) No inactive management server node found 2025-06-29 04:33:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6401d643]) (logid:e329bd2c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:33:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:33:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71fd26ee]) (logid:956cfe93) No inactive management server node found 2025-06-29 04:33:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71fd26ee]) (logid:956cfe93) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:46,552 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c156d572]) (logid:d100efc1) HA health check task is running... 2025-06-29 04:33:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:33:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:33:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:33:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:33:47,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:33:47,088 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:33:47,090 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:33:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:33:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f8d451]) (logid:085b5531) No inactive management server node found 2025-06-29 04:33:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8f8d451]) (logid:085b5531) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:48,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0da5cc9c]) (logid:3094f34d) Resetting hosts suitable for reconnect 2025-06-29 04:33:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0da5cc9c]) (logid:3094f34d) Completed resetting hosts suitable for reconnect 2025-06-29 04:33:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0da5cc9c]) (logid:3094f34d) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:33:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0da5cc9c]) (logid:3094f34d) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:33:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0da5cc9c]) (logid:3094f34d) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:33:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0da5cc9c]) (logid:3094f34d) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:33:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d9c88d8c]) (logid:8db8cd98) VM scheduler.poll is being called at 2025-06-29T02:34:00+0000 2025-06-29 04:33:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d9c88d8c]) (logid:8db8cd98) Got 0 scheduled jobs to be executed at 2025-06-29T02:34:00+0000 2025-06-29 04:33:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-d9c88d8c]) (logid:8db8cd98) Cleaned up 0 VM scheduled job entries 2025-06-29 04:33:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-88cabf9f]) (logid:ff917910) VM Operation Thread Running 2025-06-29 04:33:48,350 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2d249b47]) (logid:78886770) ClusterDRS.poll is being called at 2025-06-29T02:34:00+0000 2025-06-29 04:33:48,356 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-2d249b47]) (logid:78886770) Removed 0 old drs migration plans 2025-06-29 04:33:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e17359d]) (logid:788d9234) No inactive management server node found 2025-06-29 04:33:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e17359d]) (logid:788d9234) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:50,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8666: Seq 7-8666: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:33:50,371 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8666 2025-06-29 04:33:50,553 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-fa1e3dff]) (logid:76b90db4) HA health check task is running... 2025-06-29 04:33:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e2254ba]) (logid:113086e6) No inactive management server node found 2025-06-29 04:33:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e2254ba]) (logid:113086e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ffd18d7]) (logid:d54bf793) No inactive management server node found 2025-06-29 04:33:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ffd18d7]) (logid:d54bf793) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a283c696]) (logid:34bb70d7) No inactive management server node found 2025-06-29 04:33:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a283c696]) (logid:34bb70d7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:54,554 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6635363f]) (logid:26f2db61) HA health check task is running... 2025-06-29 04:33:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddf20531]) (logid:f2e2c458) No inactive management server node found 2025-06-29 04:33:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ddf20531]) (logid:f2e2c458) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcf990d5]) (logid:b80c8167) No inactive management server node found 2025-06-29 04:33:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcf990d5]) (logid:b80c8167) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a58b99c5]) (logid:586ad507) Enabled non-edge zones available for scan: 1 2025-06-29 04:33:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a58b99c5]) (logid:586ad507) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:33:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a58b99c5]) (logid:586ad507) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:33:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-03849d11]) (logid:e9affd57) Enabled non-edge zones available for scan: 1 2025-06-29 04:33:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-03849d11]) (logid:e9affd57) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:33:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e304b41]) (logid:75887a04) No inactive management server node found 2025-06-29 04:33:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e304b41]) (logid:75887a04) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:33:58,556 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-26900f8f]) (logid:b2c5ecd4) HA health check task is running... 2025-06-29 04:33:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7220bb13]) (logid:b034edda) No inactive management server node found 2025-06-29 04:33:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7220bb13]) (logid:b034edda) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:00,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8667: Seq 7-8667: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:00,368 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8667 2025-06-29 04:34:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd2a2901]) (logid:330dbb82) No inactive management server node found 2025-06-29 04:34:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd2a2901]) (logid:330dbb82) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:02,557 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-21e50fe6]) (logid:b1814056) HA health check task is running... 2025-06-29 04:34:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6424fb75]) (logid:728ce102) No inactive management server node found 2025-06-29 04:34:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6424fb75]) (logid:728ce102) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-121d826f]) (logid:a537d8c7) No inactive management server node found 2025-06-29 04:34:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-121d826f]) (logid:a537d8c7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-940bc739]) (logid:e35a5686) No inactive management server node found 2025-06-29 04:34:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-940bc739]) (logid:e35a5686) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:06,558 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1a269dbd]) (logid:2c78b11c) HA health check task is running... 2025-06-29 04:34:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fffc0431]) (logid:3fd32053) No inactive management server node found 2025-06-29 04:34:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fffc0431]) (logid:3fd32053) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e57523ee]) (logid:ded8222d) No inactive management server node found 2025-06-29 04:34:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e57523ee]) (logid:ded8222d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13d3ca54]) (logid:7628c3f6) No inactive management server node found 2025-06-29 04:34:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13d3ca54]) (logid:7628c3f6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:10,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8668: Seq 7-8668: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:10,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8668 2025-06-29 04:34:10,560 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0fa35c23]) (logid:7366df50) HA health check task is running... 2025-06-29 04:34:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88776c17]) (logid:25311499) No inactive management server node found 2025-06-29 04:34:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88776c17]) (logid:25311499) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2e53c79b]) (logid:38525c44) Found 0 routers to update status. 2025-06-29 04:34:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2e53c79b]) (logid:38525c44) Found 0 VPC's to update Redundant State. 2025-06-29 04:34:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2e53c79b]) (logid:38525c44) Found 0 networks to update RvR status. 2025-06-29 04:34:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43d339a1]) (logid:0dc3197b) No inactive management server node found 2025-06-29 04:34:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43d339a1]) (logid:0dc3197b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:14,561 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f1d17751]) (logid:c71ce921) HA health check task is running... 2025-06-29 04:34:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ad4f8fe]) (logid:18cb4f53) No inactive management server node found 2025-06-29 04:34:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ad4f8fe]) (logid:18cb4f53) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:15,359 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8669: Seq 7-8669: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:15,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8669 2025-06-29 04:34:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-abc5fa5e]) (logid:6f573705) No inactive management server node found 2025-06-29 04:34:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-abc5fa5e]) (logid:6f573705) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92c4d352]) (logid:a80a9490) No inactive management server node found 2025-06-29 04:34:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92c4d352]) (logid:a80a9490) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:18,050 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-9dee2b69]) (logid:08d93989) AutoScaling Monitor is running... 2025-06-29 04:34:18,121 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-98a6e3f1]) (logid:94c3834e) Starting to remove all volume_stats rows older than [Sat Jun 28 16:34:18 CEST 2025]. 2025-06-29 04:34:18,121 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-98a6e3f1]) (logid:94c3834e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:34:18 CEST 2025]. 2025-06-29 04:34:18,562 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cfc684d0]) (logid:958cafef) HA health check task is running... 2025-06-29 04:34:18,819 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-bee6870c]) (logid:187ae6cf) Starting to remove all vm_stats rows older than [Sat Jun 28 16:34:18 CEST 2025]. 2025-06-29 04:34:18,848 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-bee6870c]) (logid:187ae6cf) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:34:18 CEST 2025]. 2025-06-29 04:34:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e7c3fd1]) (logid:3f763529) No inactive management server node found 2025-06-29 04:34:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e7c3fd1]) (logid:3f763529) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:20,713 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:34:20,715 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:34:20,717 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) Seq 2-4725683383995203828: Routed from 90520738888109 2025-06-29 04:34:20,761 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) Seq 1-4725683383995203828: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:34:20,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:34:20,762 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) Seq 2-3926857400090362119: Routed from 90520738888109 2025-06-29 04:34:20,807 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-8dfbcfed]) (logid:fbfc6afc) Seq 2-3926857400090362119: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:34:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f492863]) (logid:20eb69bc) No inactive management server node found 2025-06-29 04:34:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f492863]) (logid:20eb69bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:34:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-648b6afb]) (logid:7c65c1e0) No inactive management server node found 2025-06-29 04:34:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-648b6afb]) (logid:7c65c1e0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:22,563 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-5ad49ca5]) (logid:28ea106d) HA health check task is running... 2025-06-29 04:34:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-007c6274]) (logid:6a80a57a) No inactive management server node found 2025-06-29 04:34:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-007c6274]) (logid:6a80a57a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:25,279 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:34:25,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:34:25,283 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) Seq 2-4725683383995203829: Routed from 90520738888109 2025-06-29 04:34:25,337 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) Seq 1-4725683383995203829: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:34:25,360 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8670: Seq 7-8670: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:25,365 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85a57c99]) (logid:658cc01a) No inactive management server node found 2025-06-29 04:34:25,365 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85a57c99]) (logid:658cc01a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:25,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8670 2025-06-29 04:34:25,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:34:25,379 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) Seq 2-3926857400090362120: Routed from 90520738888109 2025-06-29 04:34:25,496 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-4b2b5c95]) (logid:3d72d6f1) Seq 2-3926857400090362120: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:34:26,547 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) StorageCollector is running... 2025-06-29 04:34:26,553 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:34:26,553 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:34:26,553 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:34:26,554 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3938116399158788161: Routed from 90520738888109 2025-06-29 04:34:26,565 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3ae21117]) (logid:7b96e954) HA health check task is running... 2025-06-29 04:34:26,610 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 4-3938116399158788161: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:34:26,613 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:34:26,613 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:34:26,613 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:34:26,614 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3926857400090362121: Routed from 90520738888109 2025-06-29 04:34:26,626 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3926857400090362121: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:34:26,636 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:34:26,637 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:34:26,637 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:34:26,637 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3926857400090362122: Routed from 90520738888109 2025-06-29 04:34:26,730 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3926857400090362122: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:34:26,741 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:34:26,741 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:34:26,741 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:34:26,742 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-4725683383995203830: Routed from 90520738888109 2025-06-29 04:34:26,785 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 1-4725683383995203830: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:34:26,787 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:34:26,787 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:34:26,787 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:34:26,788 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3926857400090362123: Routed from 90520738888109 2025-06-29 04:34:26,834 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d6e9a58d]) (logid:32f100b1) Seq 2-3926857400090362123: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:34:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2f9cf17]) (logid:bf072c00) No inactive management server node found 2025-06-29 04:34:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2f9cf17]) (logid:bf072c00) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-77b86195]) (logid:860f7579) Enabled non-edge zones available for scan: 1 2025-06-29 04:34:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-77b86195]) (logid:860f7579) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:34:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-77b86195]) (logid:860f7579) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:34:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-08338b95]) (logid:6a850f4a) Enabled non-edge zones available for scan: 1 2025-06-29 04:34:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-08338b95]) (logid:6a850f4a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:34:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96c30271]) (logid:ba95dbd4) No inactive management server node found 2025-06-29 04:34:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96c30271]) (logid:ba95dbd4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aae43378]) (logid:5e2814ed) No inactive management server node found 2025-06-29 04:34:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aae43378]) (logid:5e2814ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:30,566 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-cdee0642]) (logid:7ca3b168) HA health check task is running... 2025-06-29 04:34:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf4bae4f]) (logid:b76b4661) No inactive management server node found 2025-06-29 04:34:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf4bae4f]) (logid:b76b4661) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0044f01c]) (logid:163c6abe) No inactive management server node found 2025-06-29 04:34:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0044f01c]) (logid:163c6abe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaa737ff]) (logid:43f104f9) No inactive management server node found 2025-06-29 04:34:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaa737ff]) (logid:43f104f9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:34,567 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-95233f46]) (logid:0471500b) HA health check task is running... 2025-06-29 04:34:35,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8671: Seq 7-8671: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:35,379 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8671 2025-06-29 04:34:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8083e7bf]) (logid:5a5f5141) No inactive management server node found 2025-06-29 04:34:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8083e7bf]) (logid:5a5f5141) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebbdb6a7]) (logid:7f008b73) No inactive management server node found 2025-06-29 04:34:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebbdb6a7]) (logid:7f008b73) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:38,569 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e03566cd]) (logid:6eb28e04) HA health check task is running... 2025-06-29 04:34:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e09afad]) (logid:a68b4b90) No inactive management server node found 2025-06-29 04:34:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6e09afad]) (logid:a68b4b90) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6405a6fb]) (logid:67e0e726) No inactive management server node found 2025-06-29 04:34:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6405a6fb]) (logid:67e0e726) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45243169]) (logid:3cce7aea) No inactive management server node found 2025-06-29 04:34:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45243169]) (logid:3cce7aea) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:42,570 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ee7bdb3f]) (logid:13662ef0) HA health check task is running... 2025-06-29 04:34:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-94e2b8f3]) (logid:12ece198) DbCollector is running... 2025-06-29 04:34:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fa287622]) (logid:16271291) Found 0 routers to update status. 2025-06-29 04:34:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fa287622]) (logid:16271291) Found 0 VPC's to update Redundant State. 2025-06-29 04:34:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fa287622]) (logid:16271291) Found 0 networks to update RvR status. 2025-06-29 04:34:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aaeaba4]) (logid:d339db31) No inactive management server node found 2025-06-29 04:34:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9aaeaba4]) (logid:d339db31) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) ManagementServerCollector is running... 2025-06-29 04:34:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) DB properties were already loaded 2025-06-29 04:34:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:34:44,054 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Execution of process [4067326] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:34:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Exit value of process [4067326] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:34:44,055 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Process [4067326] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:34:44,055 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) usage install: null 2025-06-29 04:34:44,055 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Sessions active: 0 2025-06-29 04:34:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:34:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067329] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:34:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:34:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067334] for command [/bin/bash -c uname -r ]. 2025-06-29 04:34:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:34:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067335] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:34:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) system memory from /proc: 4887515136 2025-06-29 04:34:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:34:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067340] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:34:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) free memory from /proc: 328093696 2025-06-29 04:34:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:34:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067345] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:34:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) used memory from /proc: 1024744 2025-06-29 04:34:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:34:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067346] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:34:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:34:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067347] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:34:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:34:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067348] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:34:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:34:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067355] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:34:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:34:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Successfully executed process [4067356] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail buffersdirect.capacity, 321668 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail buffersdirect.count, 93 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail buffersdirect.used, 321668 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail buffersmapped.capacity, 0 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail buffersmapped.count, 0 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail buffersmapped.used, 0 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail jvmuptime, 3543830 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memoryheap.committed, 638058496 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memoryheap.init, 77594624 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memoryheap.usage, 0.235082700206041 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorynon-heap.committed, 198311936 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorynon-heap.max, -1 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorynon-heap.usage, -1.93279552E8 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorynon-heap.used, 193279552 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11303235803785658 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21073416202885994 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.Metaspace.usage, 0.9929311704792548 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.PS-Eden-Space.usage, 0.4005588015334142 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:34:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:34:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorytotal.committed, 836370432 2025-06-29 04:34:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorytotal.init, 85262336 2025-06-29 04:34:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorytotal.max, 1908932607 2025-06-29 04:34:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail memorytotal.used, 642036584 2025-06-29 04:34:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail threadsdeadlocks, [] 2025-06-29 04:34:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail threadsnew.count, 0 2025-06-29 04:34:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:34:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-795e43a8]) (logid:c7e43f8a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:34:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":190001520,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3543758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":641336528,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":448757032,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":328093696,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.03,0.16],"systemCyclesUsage":[66418051,53837988,4589968607],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:34:44,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 117, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:34:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":190001520,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3543758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":641336528,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":448757032,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":328093696,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.03,0.16],"systemCyclesUsage":[66418051,53837988,4589968607],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:34:44,131 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:34:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:34:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-25:[ctx-b4abe2c0]) (logid:cad44c71) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:34:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":190001520,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3543758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":641336528,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":448757032,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":328093696,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.03,0.16],"systemCyclesUsage":[66418051,53837988,4589968607],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:34:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 117, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:34:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":190001520,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3543758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.05,"totalInit":85262336,"totalUsed":641336528,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":448757032,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":328093696,"systemMemoryUsed":1024744,"systemMemoryVirtualSize":6219411456,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.05,0.03,0.16],"systemCyclesUsage":[66418051,53837988,4589968607],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:34:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d64bb503]) (logid:6518911d) No inactive management server node found 2025-06-29 04:34:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d64bb503]) (logid:6518911d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:45,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8672: Seq 7-8672: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:45,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8672 2025-06-29 04:34:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:34:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:34:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10b0a288]) (logid:676c1619) No inactive management server node found 2025-06-29 04:34:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10b0a288]) (logid:676c1619) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:46,571 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ed63d600]) (logid:8a31459d) HA health check task is running... 2025-06-29 04:34:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:34:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:34:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:34:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:34:47,072 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:34:47,074 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:34:47,075 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:34:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:34:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-087cb8f4]) (logid:f464fc57) No inactive management server node found 2025-06-29 04:34:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-087cb8f4]) (logid:f464fc57) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bd333ebc]) (logid:cf9383e5) VM scheduler.poll is being called at 2025-06-29T02:35:00+0000 2025-06-29 04:34:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bd333ebc]) (logid:cf9383e5) Got 0 scheduled jobs to be executed at 2025-06-29T02:35:00+0000 2025-06-29 04:34:48,160 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bd333ebc]) (logid:cf9383e5) Cleaned up 0 VM scheduled job entries 2025-06-29 04:34:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-46338daf]) (logid:7713869f) VM Operation Thread Running 2025-06-29 04:34:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e7e027c2]) (logid:644851f9) ClusterDRS.poll is being called at 2025-06-29T02:35:00+0000 2025-06-29 04:34:48,357 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e7e027c2]) (logid:644851f9) Removed 0 old drs migration plans 2025-06-29 04:34:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfbcf34e]) (logid:0d6fa07e) No inactive management server node found 2025-06-29 04:34:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfbcf34e]) (logid:0d6fa07e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:50,573 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4b575005]) (logid:6c2859f9) HA health check task is running... 2025-06-29 04:34:50,857 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6aaa8a7]) (logid:5d8b1ac1) No inactive management server node found 2025-06-29 04:34:50,857 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6aaa8a7]) (logid:5d8b1ac1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:52,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0e54433]) (logid:17d084b1) No inactive management server node found 2025-06-29 04:34:52,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0e54433]) (logid:17d084b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-527ee68e]) (logid:7802982b) No inactive management server node found 2025-06-29 04:34:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-527ee68e]) (logid:7802982b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:54,574 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b7a6ea09]) (logid:43fa5779) HA health check task is running... 2025-06-29 04:34:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-643218b0]) (logid:af47127f) No inactive management server node found 2025-06-29 04:34:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-643218b0]) (logid:af47127f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:55,361 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8674: Seq 7-8674: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:34:55,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8674 2025-06-29 04:34:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-902a970b]) (logid:96803fb5) No inactive management server node found 2025-06-29 04:34:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-902a970b]) (logid:96803fb5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-678e8ce8]) (logid:55c60692) Enabled non-edge zones available for scan: 1 2025-06-29 04:34:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-678e8ce8]) (logid:55c60692) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:34:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-678e8ce8]) (logid:55c60692) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:34:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0e309ba5]) (logid:766a2bd6) Enabled non-edge zones available for scan: 1 2025-06-29 04:34:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0e309ba5]) (logid:766a2bd6) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:34:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-669b0578]) (logid:20008038) No inactive management server node found 2025-06-29 04:34:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-669b0578]) (logid:20008038) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:34:58,576 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-37696b80]) (logid:da0f6688) HA health check task is running... 2025-06-29 04:34:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab1d0a9e]) (logid:ec3c1cf9) No inactive management server node found 2025-06-29 04:34:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab1d0a9e]) (logid:ec3c1cf9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97c00f3c]) (logid:c7c70807) No inactive management server node found 2025-06-29 04:35:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97c00f3c]) (logid:c7c70807) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:02,577 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-471aea54]) (logid:d81243ce) HA health check task is running... 2025-06-29 04:35:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96cd1ea8]) (logid:e67c818f) No inactive management server node found 2025-06-29 04:35:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96cd1ea8]) (logid:e67c818f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-312430c0]) (logid:2142a1f5) No inactive management server node found 2025-06-29 04:35:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-312430c0]) (logid:2142a1f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:05,363 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8675: Seq 7-8675: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:35:05,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8675 2025-06-29 04:35:05,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6747b4b]) (logid:bcd82ebe) No inactive management server node found 2025-06-29 04:35:05,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6747b4b]) (logid:bcd82ebe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:06,578 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-163097f9]) (logid:33ccc9e0) HA health check task is running... 2025-06-29 04:35:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fa34ae0]) (logid:7ae6c37c) No inactive management server node found 2025-06-29 04:35:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fa34ae0]) (logid:7ae6c37c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:08,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12ca0dc3]) (logid:e138e183) No inactive management server node found 2025-06-29 04:35:08,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12ca0dc3]) (logid:e138e183) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c39575]) (logid:ddcca4ce) No inactive management server node found 2025-06-29 04:35:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c39575]) (logid:ddcca4ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:10,580 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e71a9b41]) (logid:ed83b8eb) HA health check task is running... 2025-06-29 04:35:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59bac2fd]) (logid:e5ff21dd) No inactive management server node found 2025-06-29 04:35:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59bac2fd]) (logid:e5ff21dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5ab08cf]) (logid:972c64d1) Found 0 routers to update status. 2025-06-29 04:35:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5ab08cf]) (logid:972c64d1) Found 0 VPC's to update Redundant State. 2025-06-29 04:35:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f5ab08cf]) (logid:972c64d1) Found 0 networks to update RvR status. 2025-06-29 04:35:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bf39d1e]) (logid:208962c2) No inactive management server node found 2025-06-29 04:35:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2bf39d1e]) (logid:208962c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:14,581 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-cc0f9d73]) (logid:26bfedb4) HA health check task is running... 2025-06-29 04:35:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6368f173]) (logid:ccdd8e72) No inactive management server node found 2025-06-29 04:35:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6368f173]) (logid:ccdd8e72) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:15,362 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8676: Seq 7-8676: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:35:15,372 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8676 2025-06-29 04:35:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa37ef06]) (logid:bb1e03ca) No inactive management server node found 2025-06-29 04:35:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa37ef06]) (logid:bb1e03ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f524c3d0]) (logid:8142da2f) No inactive management server node found 2025-06-29 04:35:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f524c3d0]) (logid:8142da2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:18,052 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-98df9d89]) (logid:954381c2) AutoScaling Monitor is running... 2025-06-29 04:35:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-df0fc4f1]) (logid:2d9b0dff) Resetting hosts suitable for reconnect 2025-06-29 04:35:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-df0fc4f1]) (logid:2d9b0dff) Completed resetting hosts suitable for reconnect 2025-06-29 04:35:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-df0fc4f1]) (logid:2d9b0dff) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:35:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-df0fc4f1]) (logid:2d9b0dff) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:35:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-df0fc4f1]) (logid:2d9b0dff) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:35:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-df0fc4f1]) (logid:2d9b0dff) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:35:18,124 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-004d42fa]) (logid:86a07ea6) Starting to remove all volume_stats rows older than [Sat Jun 28 16:35:18 CEST 2025]. 2025-06-29 04:35:18,124 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-004d42fa]) (logid:86a07ea6) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:35:18 CEST 2025]. 2025-06-29 04:35:18,583 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c97fdb6a]) (logid:2fa8c1a4) HA health check task is running... 2025-06-29 04:35:18,850 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-7d434969]) (logid:e7b2dc70) Starting to remove all vm_stats rows older than [Sat Jun 28 16:35:18 CEST 2025]. 2025-06-29 04:35:18,876 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-7d434969]) (logid:e7b2dc70) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:35:18 CEST 2025]. 2025-06-29 04:35:19,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b4e60a]) (logid:127a8618) No inactive management server node found 2025-06-29 04:35:19,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b4e60a]) (logid:127a8618) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:20,810 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:35:20,812 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:35:20,814 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) Seq 2-4725683383995203831: Routed from 90520738888109 2025-06-29 04:35:20,856 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) Seq 1-4725683383995203831: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:35:20,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:35:20,859 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) Seq 2-3926857400090362124: Routed from 90520738888109 2025-06-29 04:35:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62228e8b]) (logid:c0c01c01) No inactive management server node found 2025-06-29 04:35:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62228e8b]) (logid:c0c01c01) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:20,904 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-f275a737]) (logid:ecbb21b9) Seq 2-3926857400090362124: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:35:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77743d41]) (logid:5a55aa48) No inactive management server node found 2025-06-29 04:35:22,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77743d41]) (logid:5a55aa48) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:35:22,584 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-299d722a]) (logid:790f3d80) HA health check task is running... 2025-06-29 04:35:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb158b9a]) (logid:986521d0) No inactive management server node found 2025-06-29 04:35:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb158b9a]) (logid:986521d0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b62bfa11]) (logid:343847bc) No inactive management server node found 2025-06-29 04:35:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b62bfa11]) (logid:343847bc) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:35:25,364 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8677: Seq 7-8677: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:35:25,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8677 2025-06-29 04:35:25,550 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:35:25,555 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:35:25,556 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) Seq 2-4725683383995203832: Routed from 90520738888109 2025-06-29 04:35:25,609 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) Seq 1-4725683383995203832: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:35:25,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:35:25,644 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) Seq 2-3926857400090362125: Routed from 90520738888109 2025-06-29 04:35:25,733 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-c8810ff0]) (logid:82145d2e) Seq 2-3926857400090362125: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:35:26,585 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-7bb4be51]) (logid:5a15efd5) HA health check task is running... 2025-06-29 04:35:26,834 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) StorageCollector is running... 2025-06-29 04:35:26,840 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:35:26,840 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:35:26,840 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:35:26,841 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-3938116399158788162: Routed from 90520738888109 2025-06-29 04:35:26,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d3d1656]) (logid:cb5c51aa) No inactive management server node found 2025-06-29 04:35:26,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d3d1656]) (logid:cb5c51aa) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:26,896 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 4-3938116399158788162: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:35:26,900 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:35:26,900 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:35:26,900 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:35:26,902 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-4725683383995203833: Routed from 90520738888109 2025-06-29 04:35:26,949 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 1-4725683383995203833: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:35:26,960 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:35:26,960 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:35:26,960 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:35:26,961 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-3926857400090362126: Routed from 90520738888109 2025-06-29 04:35:26,979 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-3926857400090362126: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:35:26,989 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:35:26,989 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:35:26,989 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:35:26,990 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-4725683383995203834: Routed from 90520738888109 2025-06-29 04:35:27,034 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 1-4725683383995203834: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:35:27,036 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:35:27,036 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:35:27,036 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:35:27,037 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-3926857400090362127: Routed from 90520738888109 2025-06-29 04:35:27,082 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e9d2776a]) (logid:2f83659c) Seq 2-3926857400090362127: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:35:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-622d617a]) (logid:cc8ca65d) Enabled non-edge zones available for scan: 1 2025-06-29 04:35:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-622d617a]) (logid:cc8ca65d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:35:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-622d617a]) (logid:cc8ca65d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:35:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8d41f190]) (logid:54a2a7c5) Enabled non-edge zones available for scan: 1 2025-06-29 04:35:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-8d41f190]) (logid:54a2a7c5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:35:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50cdfd76]) (logid:5c5b6587) No inactive management server node found 2025-06-29 04:35:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50cdfd76]) (logid:5c5b6587) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74e56284]) (logid:947be49e) No inactive management server node found 2025-06-29 04:35:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74e56284]) (logid:947be49e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:30,587 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-bc05e462]) (logid:8ed9c129) HA health check task is running... 2025-06-29 04:35:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-594d74ea]) (logid:1193695a) No inactive management server node found 2025-06-29 04:35:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-594d74ea]) (logid:1193695a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-978a0b35]) (logid:2cf46e32) No inactive management server node found 2025-06-29 04:35:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-978a0b35]) (logid:2cf46e32) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:34,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0e87c79]) (logid:f7cb3d93) No inactive management server node found 2025-06-29 04:35:34,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0e87c79]) (logid:f7cb3d93) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:34,588 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1ecd51d2]) (logid:19aa39f8) HA health check task is running... 2025-06-29 04:35:35,364 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8678: Seq 7-8678: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:35:35,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8678 2025-06-29 04:35:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc380d66]) (logid:b9815c44) No inactive management server node found 2025-06-29 04:35:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc380d66]) (logid:b9815c44) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73afaffc]) (logid:5b67f499) No inactive management server node found 2025-06-29 04:35:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73afaffc]) (logid:5b67f499) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:38,590 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d34bcd32]) (logid:f2861009) HA health check task is running... 2025-06-29 04:35:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e51b6d1]) (logid:fac7db64) No inactive management server node found 2025-06-29 04:35:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e51b6d1]) (logid:fac7db64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64940e18]) (logid:cc93c012) No inactive management server node found 2025-06-29 04:35:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64940e18]) (logid:cc93c012) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f416a27]) (logid:0221ce87) No inactive management server node found 2025-06-29 04:35:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f416a27]) (logid:0221ce87) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:42,591 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-25c28ada]) (logid:a77f85f3) HA health check task is running... 2025-06-29 04:35:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-e2c00a75]) (logid:1e7c470e) DbCollector is running... 2025-06-29 04:35:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-12d8c6aa]) (logid:1c03e9bf) Found 0 routers to update status. 2025-06-29 04:35:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-12d8c6aa]) (logid:1c03e9bf) Found 0 VPC's to update Redundant State. 2025-06-29 04:35:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-12d8c6aa]) (logid:1c03e9bf) Found 0 networks to update RvR status. 2025-06-29 04:35:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51719c70]) (logid:04587195) No inactive management server node found 2025-06-29 04:35:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51719c70]) (logid:04587195) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) ManagementServerCollector is running... 2025-06-29 04:35:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) DB properties were already loaded 2025-06-29 04:35:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:35:44,054 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Execution of process [4067374] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:35:44,054 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Exit value of process [4067374] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:35:44,055 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Process [4067374] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:35:44,055 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) usage install: null 2025-06-29 04:35:44,055 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Sessions active: 0 2025-06-29 04:35:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:35:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067377] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:35:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:35:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067382] for command [/bin/bash -c uname -r ]. 2025-06-29 04:35:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:35:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067383] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:35:44,072 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) system memory from /proc: 4887515136 2025-06-29 04:35:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:35:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067388] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:35:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) free memory from /proc: 327729152 2025-06-29 04:35:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:35:44,089 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067393] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:35:44,089 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) used memory from /proc: 1025128 2025-06-29 04:35:44,089 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:35:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067394] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:35:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:35:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067395] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:35:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:35:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067396] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:35:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:35:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067403] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:35:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:35:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Successfully executed process [4067404] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail buffersdirect.capacity, 323303 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail buffersdirect.count, 95 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail buffersdirect.used, 323303 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail buffersmapped.capacity, 0 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail buffersmapped.count, 0 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail buffersmapped.used, 0 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail gcPS-Scavenge.count, 108 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail gcPS-Scavenge.time, 559 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail jvmuptime, 3603829 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:35:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memoryheap.committed, 638058496 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memoryheap.init, 77594624 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memoryheap.usage, 0.24869753809559317 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorynon-heap.committed, 198311936 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorynon-heap.max, -1 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorynon-heap.usage, -1.93526968E8 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorynon-heap.used, 193526968 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11349890029325513 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21190784650248276 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.Compressed-Class-Space.usage, 0.012385323643684387 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.Metaspace.usage, 0.99325984780868 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.PS-Eden-Space.usage, 0.44024785257893817 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.PS-Old-Gen.usage, 0.12613762066679443 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorypools.PS-Survivor-Space.usage, 0.929693857828776 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorytotal.committed, 836370432 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorytotal.init, 85262336 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorytotal.max, 1908932607 2025-06-29 04:35:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail memorytotal.used, 668273808 2025-06-29 04:35:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail threadsdeadlocks, [] 2025-06-29 04:35:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail threadsnew.count, 0 2025-06-29 04:35:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) not storing detail threadstimed_waiting.count, 85 2025-06-29 04:35:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-c68c08ac]) (logid:16c02b7a) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:35:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":164023280,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3603758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":667562184,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":474746840,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":327729152,"systemMemoryUsed":1025128,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.04,0.16],"systemCyclesUsage":[66418444,53838326,4589991628],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:35:44,129 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 119, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:35:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":164023280,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3603758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":667562184,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":474746840,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":327729152,"systemMemoryUsed":1025128,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.04,0.16],"systemCyclesUsage":[66418444,53838326,4589991628],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:35:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:35:44,137 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-25:[ctx-f38d7f80]) (logid:71859ea7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:35:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":164023280,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3603758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":667562184,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":474746840,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":327729152,"systemMemoryUsed":1025128,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.04,0.16],"systemCyclesUsage":[66418444,53838326,4589991628],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:35:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:35:44,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 119, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:35:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":638058496,"freeJvmMemoryBytes":164023280,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3603758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":667562184,"totalCommitted":836370432,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":474746840,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":327729152,"systemMemoryUsed":1025128,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.04,0.16],"systemCyclesUsage":[66418444,53838326,4589991628],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:35:44,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e3f838d]) (logid:c52059dc) No inactive management server node found 2025-06-29 04:35:44,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2e3f838d]) (logid:c52059dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:45,363 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8679: Seq 7-8679: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:35:45,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8679 2025-06-29 04:35:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:35:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:35:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df5a66b5]) (logid:97cafbae) No inactive management server node found 2025-06-29 04:35:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df5a66b5]) (logid:97cafbae) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:46,592 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6750975a]) (logid:5739fd91) HA health check task is running... 2025-06-29 04:35:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:35:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:35:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:35:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:35:47,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:35:47,082 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:35:47,083 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:35:47,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:35:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cff7ed0a]) (logid:f0dfe528) No inactive management server node found 2025-06-29 04:35:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cff7ed0a]) (logid:f0dfe528) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-780c1216]) (logid:421d33b2) VM scheduler.poll is being called at 2025-06-29T02:36:00+0000 2025-06-29 04:35:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-780c1216]) (logid:421d33b2) Got 0 scheduled jobs to be executed at 2025-06-29T02:36:00+0000 2025-06-29 04:35:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-780c1216]) (logid:421d33b2) Cleaned up 0 VM scheduled job entries 2025-06-29 04:35:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9143a41b]) (logid:88d3d7c2) VM Operation Thread Running 2025-06-29 04:35:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6df35e5b]) (logid:ad45f153) ClusterDRS.poll is being called at 2025-06-29T02:36:00+0000 2025-06-29 04:35:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6df35e5b]) (logid:ad45f153) Removed 0 old drs migration plans 2025-06-29 04:35:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bc2b878]) (logid:e8a12231) No inactive management server node found 2025-06-29 04:35:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bc2b878]) (logid:e8a12231) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:50,594 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-87849e88]) (logid:c0e64805) HA health check task is running... 2025-06-29 04:35:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf136d60]) (logid:3a7a2c4a) No inactive management server node found 2025-06-29 04:35:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf136d60]) (logid:3a7a2c4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9e0acc2]) (logid:57ace642) No inactive management server node found 2025-06-29 04:35:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9e0acc2]) (logid:57ace642) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00179301]) (logid:37d4f3ba) No inactive management server node found 2025-06-29 04:35:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-00179301]) (logid:37d4f3ba) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:35:54,595 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-103c25ef]) (logid:8b90e7de) HA health check task is running... 2025-06-29 04:35:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acce7bc4]) (logid:ab22bf4b) No inactive management server node found 2025-06-29 04:35:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acce7bc4]) (logid:ab22bf4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:55,364 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8681: Seq 7-8681: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:35:55,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8681 2025-06-29 04:35:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e2468cf]) (logid:eb52be1c) No inactive management server node found 2025-06-29 04:35:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e2468cf]) (logid:eb52be1c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:58,091 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fa926438]) (logid:c902887b) Enabled non-edge zones available for scan: 1 2025-06-29 04:35:58,099 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-fa926438]) (logid:c902887b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:35:58,102 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fa926438]) (logid:c902887b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:35:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-42b67bc8]) (logid:46620810) Enabled non-edge zones available for scan: 1 2025-06-29 04:35:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-42b67bc8]) (logid:46620810) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:35:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71caffaa]) (logid:e40ca536) No inactive management server node found 2025-06-29 04:35:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71caffaa]) (logid:e40ca536) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:35:58,597 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9e8e17cd]) (logid:d4901240) HA health check task is running... 2025-06-29 04:35:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c50990e1]) (logid:0d323fe1) No inactive management server node found 2025-06-29 04:35:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c50990e1]) (logid:0d323fe1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecfa564b]) (logid:5eea1d69) No inactive management server node found 2025-06-29 04:36:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecfa564b]) (logid:5eea1d69) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:02,598 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2998eadb]) (logid:a28a4c47) HA health check task is running... 2025-06-29 04:36:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e49d8e8]) (logid:83048057) No inactive management server node found 2025-06-29 04:36:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e49d8e8]) (logid:83048057) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78edb7df]) (logid:5bb1d406) No inactive management server node found 2025-06-29 04:36:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78edb7df]) (logid:5bb1d406) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:05,374 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8682: Seq 7-8682: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:05,385 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8682 2025-06-29 04:36:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3313fb0]) (logid:c1758d2e) No inactive management server node found 2025-06-29 04:36:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3313fb0]) (logid:c1758d2e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:06,600 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1f538018]) (logid:25e17544) HA health check task is running... 2025-06-29 04:36:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43397b55]) (logid:76126b3d) No inactive management server node found 2025-06-29 04:36:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43397b55]) (logid:76126b3d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fa79577]) (logid:6aa3f809) No inactive management server node found 2025-06-29 04:36:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fa79577]) (logid:6aa3f809) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-391961c6]) (logid:bf6d82b9) No inactive management server node found 2025-06-29 04:36:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-391961c6]) (logid:bf6d82b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:10,601 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-45cb1ba9]) (logid:6aa96e50) HA health check task is running... 2025-06-29 04:36:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac441bb0]) (logid:fd2fd34a) No inactive management server node found 2025-06-29 04:36:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac441bb0]) (logid:fd2fd34a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e10b38a3]) (logid:85a29652) Found 0 routers to update status. 2025-06-29 04:36:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e10b38a3]) (logid:85a29652) Found 0 VPC's to update Redundant State. 2025-06-29 04:36:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e10b38a3]) (logid:85a29652) Found 0 networks to update RvR status. 2025-06-29 04:36:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a8763ab]) (logid:64634041) No inactive management server node found 2025-06-29 04:36:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4a8763ab]) (logid:64634041) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:14,602 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2c966e74]) (logid:9200abed) HA health check task is running... 2025-06-29 04:36:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7724478d]) (logid:ae7e8d66) No inactive management server node found 2025-06-29 04:36:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7724478d]) (logid:ae7e8d66) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:15,364 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8683: Seq 7-8683: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:15,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8683 2025-06-29 04:36:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91b4046c]) (logid:99c810fd) No inactive management server node found 2025-06-29 04:36:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91b4046c]) (logid:99c810fd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da42d5b9]) (logid:9e945452) No inactive management server node found 2025-06-29 04:36:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-da42d5b9]) (logid:9e945452) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:18,054 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-3abc26a7]) (logid:bffe19f1) AutoScaling Monitor is running... 2025-06-29 04:36:18,126 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-fac79ce7]) (logid:ce1b17d9) Starting to remove all volume_stats rows older than [Sat Jun 28 16:36:18 CEST 2025]. 2025-06-29 04:36:18,127 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-fac79ce7]) (logid:ce1b17d9) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:36:18 CEST 2025]. 2025-06-29 04:36:18,603 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5c221995]) (logid:1e4f4062) HA health check task is running... 2025-06-29 04:36:18,877 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-bbf7c28c]) (logid:cb7ee926) Starting to remove all vm_stats rows older than [Sat Jun 28 16:36:18 CEST 2025]. 2025-06-29 04:36:18,901 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-bbf7c28c]) (logid:cb7ee926) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:36:18 CEST 2025]. 2025-06-29 04:36:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35cd400f]) (logid:c059163b) No inactive management server node found 2025-06-29 04:36:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35cd400f]) (logid:c059163b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-766712bc]) (logid:545101e1) No inactive management server node found 2025-06-29 04:36:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-766712bc]) (logid:545101e1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:20,907 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:36:20,909 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:36:20,910 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) Seq 2-4725683383995203835: Routed from 90520738888109 2025-06-29 04:36:20,954 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) Seq 1-4725683383995203835: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:36:20,955 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:36:20,955 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) Seq 2-3926857400090362128: Routed from 90520738888109 2025-06-29 04:36:21,000 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e3afbb81]) (logid:70a4add6) Seq 2-3926857400090362128: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:36:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-397bec46]) (logid:2d886570) No inactive management server node found 2025-06-29 04:36:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-397bec46]) (logid:2d886570) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:22,605 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-068f565a]) (logid:f055396f) HA health check task is running... 2025-06-29 04:36:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72bcf8a9]) (logid:0b5a228b) No inactive management server node found 2025-06-29 04:36:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-72bcf8a9]) (logid:0b5a228b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc8df5d2]) (logid:f9d66b60) No inactive management server node found 2025-06-29 04:36:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc8df5d2]) (logid:f9d66b60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:25,366 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8684: Seq 7-8684: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:25,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8684 2025-06-29 04:36:25,786 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:36:25,790 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:36:25,791 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) Seq 2-4725683383995203836: Routed from 90520738888109 2025-06-29 04:36:25,845 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) Seq 1-4725683383995203836: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:36:25,879 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:36:25,880 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) Seq 2-3926857400090362129: Routed from 90520738888109 2025-06-29 04:36:25,969 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-c9265403]) (logid:5fbaed68) Seq 2-3926857400090362129: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:36:26,606 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4bc8bb15]) (logid:874aa2cc) HA health check task is running... 2025-06-29 04:36:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d023343b]) (logid:4583b2ba) No inactive management server node found 2025-06-29 04:36:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d023343b]) (logid:4583b2ba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:27,082 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) StorageCollector is running... 2025-06-29 04:36:27,090 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:36:27,090 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:36:27,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:36:27,091 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3938116399158788163: Routed from 90520738888109 2025-06-29 04:36:27,147 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 4-3938116399158788163: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:36:27,151 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:36:27,151 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:36:27,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:36:27,152 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3926857400090362130: Routed from 90520738888109 2025-06-29 04:36:27,166 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3926857400090362130: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:36:27,177 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:36:27,178 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:36:27,178 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:36:27,179 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3926857400090362131: Routed from 90520738888109 2025-06-29 04:36:27,271 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3926857400090362131: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:36:27,282 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:36:27,282 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:36:27,282 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:36:27,284 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-4725683383995203837: Routed from 90520738888109 2025-06-29 04:36:27,327 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 1-4725683383995203837: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:36:27,330 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:36:27,330 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:36:27,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:36:27,331 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3926857400090362132: Routed from 90520738888109 2025-06-29 04:36:27,375 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-41ad305f]) (logid:fa0392ec) Seq 2-3926857400090362132: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:36:28,091 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-523cff59]) (logid:ad8f50e4) Enabled non-edge zones available for scan: 1 2025-06-29 04:36:28,097 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-523cff59]) (logid:ad8f50e4) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:36:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-523cff59]) (logid:ad8f50e4) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:36:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-caad4dfc]) (logid:9643f6a4) Enabled non-edge zones available for scan: 1 2025-06-29 04:36:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-caad4dfc]) (logid:9643f6a4) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:36:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-036bacd3]) (logid:fb8ecc8c) No inactive management server node found 2025-06-29 04:36:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-036bacd3]) (logid:fb8ecc8c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:29,885 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c133a55]) (logid:e8b63805) No inactive management server node found 2025-06-29 04:36:29,885 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3c133a55]) (logid:e8b63805) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:30,365 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8685: Seq 7-8685: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:30,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8685 2025-06-29 04:36:30,607 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-28a63ba4]) (logid:d3a8922e) HA health check task is running... 2025-06-29 04:36:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0c38f55]) (logid:dffeeaa2) No inactive management server node found 2025-06-29 04:36:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0c38f55]) (logid:dffeeaa2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2203c7cf]) (logid:48bb90e7) No inactive management server node found 2025-06-29 04:36:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2203c7cf]) (logid:48bb90e7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39efefab]) (logid:9e6b01e7) No inactive management server node found 2025-06-29 04:36:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39efefab]) (logid:9e6b01e7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:34,608 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-45f0ce41]) (logid:98979b4e) HA health check task is running... 2025-06-29 04:36:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf442b70]) (logid:c13a65f9) No inactive management server node found 2025-06-29 04:36:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cf442b70]) (logid:c13a65f9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91cced4b]) (logid:5c2af83d) No inactive management server node found 2025-06-29 04:36:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91cced4b]) (logid:5c2af83d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:38,610 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-040b9839]) (logid:d6720ce8) HA health check task is running... 2025-06-29 04:36:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f195030]) (logid:c355d19a) No inactive management server node found 2025-06-29 04:36:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f195030]) (logid:c355d19a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17108cfe]) (logid:1ac5d7b6) No inactive management server node found 2025-06-29 04:36:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17108cfe]) (logid:1ac5d7b6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:40,366 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8686: Seq 7-8686: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:40,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8686 2025-06-29 04:36:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd00ccc2]) (logid:e6c4a4ad) No inactive management server node found 2025-06-29 04:36:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd00ccc2]) (logid:e6c4a4ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:42,611 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a8e3dbd1]) (logid:3e79e9f7) HA health check task is running... 2025-06-29 04:36:42,866 INFO [c.c.n.v.V.VpcCleanupTask] (VpcChecker-1:[ctx-f097409a]) (logid:317fd991) Found 0 removed VPCs to cleanup 2025-06-29 04:36:43,028 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-ce6479ad]) (logid:2f89ec2b) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 04:36:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-ce2a26e5]) (logid:f8122c8e) DbCollector is running... 2025-06-29 04:36:43,081 DEBUG [c.c.a.m.A.AgentNewConnectionsMonitorTask] (NewAgentConnectionsMonitor-1:[ctx-38cbd66a]) (logid:59d4d3e6) Currently 0 active new connections, of which 0 have expired - [] 2025-06-29 04:36:43,095 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-292de77b]) (logid:29f0388c) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:36:43,096 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-292de77b]) (logid:29f0388c) Snapshot scheduler is being called at [2025-06-29T02:36:43+0000]. 2025-06-29 04:36:43,097 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-292de77b]) (logid:29f0388c) There are [0] scheduled snapshots to be executed at [2025-06-29T02:36:43+0000]. 2025-06-29 04:36:43,098 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-5714dbc3]) (logid:f5870376) Found 0 running routers. 2025-06-29 04:36:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-199b1f12]) (logid:c3d0610e) Found 0 routers to update status. 2025-06-29 04:36:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-199b1f12]) (logid:c3d0610e) Found 0 VPC's to update Redundant State. 2025-06-29 04:36:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-199b1f12]) (logid:c3d0610e) Found 0 networks to update RvR status. 2025-06-29 04:36:43,107 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-957db9e8]) (logid:46af44f4) Found 1 running routers. 2025-06-29 04:36:43,116 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-957db9e8]) (logid:46af44f4) Wait time setting on com.cloud.agent.api.routing.GetRouterAlertsCommand is 1800 seconds 2025-06-29 04:36:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-b8e2084a]) (logid:607f8062) External devices are not used. Skipping external device usage collection 2025-06-29 04:36:43,118 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-957db9e8]) (logid:46af44f4) Seq 2-3926857400090362133: Routed from 90520738888109 2025-06-29 04:36:43,118 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-957db9e8]) (logid:46af44f4) Seq 2-3926857400090362133: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterAlertsCommand":{"previousAlertTimeStamp":"2025-06-20 12:09:07","accessDetails":{"router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:43,135 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-7c6cafbf]) (logid:5d825aaf) Started resource counters recalculation periodic task. 2025-06-29 04:36:43,138 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-7c6cafbf]) (logid:5d825aaf) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:36:43,306 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-3e73c6b0]) (logid:25dd62a0) Backup backup.poll is being called at 2025-06-29T02:36:43+0000 2025-06-29 04:36:43,313 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-50e94498]) (logid:6e305d1c) checking health of usage server 2025-06-29 04:36:43,322 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-50e94498]) (logid:6e305d1c) usage server running? false, heartbeat: null 2025-06-29 04:36:43,328 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-50e94498]) (logid:6e305d1c) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 04:36:43,332 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-50e94498]) (logid:6e305d1c) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 04:36:43,366 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30f676ca]) (logid:9914111d) No inactive management server node found 2025-06-29 04:36:43,366 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30f676ca]) (logid:9914111d) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 2ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:36:43,422 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:36:43,422 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:36:43,572 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-7c6cafbf]) (logid:5d825aaf) Finished resource counters recalculation periodic task. 2025-06-29 04:36:43,671 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-3926857400090362133: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetRouterAlertsAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:43,671 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-957db9e8]) (logid:46af44f4) Seq 2-3926857400090362133: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterAlertsAnswer } } 2025-06-29 04:36:43,673 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Found 1 running routers. 2025-06-29 04:36:43,677 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:36:43,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Seq 2-3926857400090362134: Routed from 90520738888109 2025-06-29 04:36:43,679 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Seq 2-3926857400090362134: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:43,961 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 2-3926857400090362134: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:43,961 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Seq 2-3926857400090362134: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:36:43,961 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 04:36:43,964 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 04:36:44,015 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 04:36:44,017 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Seq 2-3926857400090362135: Routed from 90520738888109 2025-06-29 04:36:44,018 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Seq 2-3926857400090362135: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) ManagementServerCollector is running... 2025-06-29 04:36:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) DB properties were already loaded 2025-06-29 04:36:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:36:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Execution of process [4067425] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:36:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Exit value of process [4067425] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:36:44,058 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Process [4067425] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:36:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) usage install: null 2025-06-29 04:36:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Sessions active: 0 2025-06-29 04:36:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:36:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067428] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:36:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:36:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067433] for command [/bin/bash -c uname -r ]. 2025-06-29 04:36:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:36:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067434] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:36:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) system memory from /proc: 4887515136 2025-06-29 04:36:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:36:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067439] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:36:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) free memory from /proc: 323813376 2025-06-29 04:36:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:36:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067444] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:36:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) used memory from /proc: 1019720 2025-06-29 04:36:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:36:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067445] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:36:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:36:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067446] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:36:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:36:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067447] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:36:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:36:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067454] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:36:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:36:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Successfully executed process [4067455] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:36:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail buffersdirect.capacity, 327567 2025-06-29 04:36:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail buffersdirect.count, 98 2025-06-29 04:36:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail buffersdirect.used, 327567 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail buffersmapped.capacity, 0 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail buffersmapped.count, 0 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail buffersmapped.used, 0 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail jvmuptime, 3663832 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memoryheap.committed, 654311424 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memoryheap.init, 77594624 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memoryheap.usage, 0.1077673958409327 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorynon-heap.committed, 198574080 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorynon-heap.max, -1 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorynon-heap.usage, -1.9390308E8 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorynon-heap.used, 193903080 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11379361336976807 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.213811568967241 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.Compressed-Class-Space.usage, 0.012387551367282867 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.Metaspace.usage, 0.9926024244424883 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.PS-Eden-Space.usage, 0.019421573611251006 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorytotal.committed, 852885504 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorytotal.init, 85262336 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorytotal.max, 1908932607 2025-06-29 04:36:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail memorytotal.used, 399623776 2025-06-29 04:36:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail threadsdeadlocks, [] 2025-06-29 04:36:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail threadsnew.count, 0 2025-06-29 04:36:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) not storing detail threadstimed_waiting.count, 85 2025-06-29 04:36:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-18fd0c5f]) (logid:800562c0) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":451387056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3663761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":396863624,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205720696,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":323813376,"systemMemoryUsed":1019720,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.06,0.16],"systemCyclesUsage":[66418832,53838643,4590014672],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:36:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 121, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":451387056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3663761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":396863624,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205720696,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":323813376,"systemMemoryUsed":1019720,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.06,0.16],"systemCyclesUsage":[66418832,53838643,4590014672],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:36:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:36:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:36:44,142 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-26:[ctx-16d2ae29]) (logid:00300ccd) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":451387056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3663761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":396863624,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205720696,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":323813376,"systemMemoryUsed":1019720,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.06,0.16],"systemCyclesUsage":[66418832,53838643,4590014672],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:36:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 121, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:36:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":451387056,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3663761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.09,"totalInit":85262336,"totalUsed":396863624,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205720696,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":18,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":323813376,"systemMemoryUsed":1019720,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.09,0.06,0.16],"systemCyclesUsage":[66418832,53838643,4590014672],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:36:44,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbe26eb3]) (logid:bfdefbc3) No inactive management server node found 2025-06-29 04:36:44,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbe26eb3]) (logid:bfdefbc3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:45,416 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:[]) (logid:) Seq 2-3926857400090362135: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.5fc80f98-cdb6-45f8-a6f4-e02249e7cab2","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:45,416 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-6eeec812]) (logid:da5b4275) Seq 2-3926857400090362135: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 04:36:45,419 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 04:36:45,422 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:36:45,424 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Seq 2-3926857400090362136: Routed from 90520738888109 2025-06-29 04:36:45,424 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Seq 2-3926857400090362136: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:36:45,849 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 2-3926857400090362136: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "13.609886169433594", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "25.968074798583984", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "38.50078582763672", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "50.77338218688965", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751164561745", "lastRunDuration": "42.958974838256836", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751164561814", "lastRunDuration": "8076.171875", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751164569942", "lastRunDuration": "66.44439697265625", "message": "Memory Usage within limits with current at 50.8791%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751164570009", "lastRunDuration": "3256.709575653076", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751164573323", "lastRunDuration": "49.95608329772949", "message": "Sufficient free space is 2571.26953125 MB"}, "lastRun": {"start": "2025-06-29 02:36:01.595284", "end": "2025-06-29 02:36:13.373806", "duration": "11.778521299362183"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164201601", "lastRunDuration": "116.95194244384766", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164201718", "lastRunDuration": "119.9336051940918", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164202095", "lastRunDuration": "86.74001693725586", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164202298", "lastRunDuration": "45.577287673950195", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:30:01.598085", "end": "2025-06-29 02:30:02.400159", "duration": "0.8020739555358887"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "13.609886169433594", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "25.968074798583984", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "38.50078582763672", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "50.77338218688965", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751164561745", "lastRunDuration": "42.958974838256836", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751164561814", "lastRunDuration": "8076.171875", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751164569942", "lastRunDuration": "66.44439697265625", "message": "Memory Usage within limits with current at 50.8791%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751164570009", "lastRunDuration": "3256.709575653076", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751164573323", "lastRunDuration": "49.95608329772949", "message": "Sufficient free space is 2571.26953125 MB"}, "lastRun": {"start": "2025-06-29 02:36:01.595284", "end": "2025-06-29 02:36:13.373806", "duration": "11.778521299362183"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164201601", "lastRunDuration": "116.95194244384766", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164201718", "lastRunDuration": "119.9336051940918", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164202095", "lastRunDuration": "86.74001693725586", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164202298", "lastRunDuration": "45.577287673950195", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:30:01.598085", "end": "2025-06-29 02:30:02.400159", "duration": "0.8020739555358887"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:45,849 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Seq 2-3926857400090362136: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:36:45,849 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "13.609886169433594", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "25.968074798583984", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "38.50078582763672", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "50.77338218688965", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751164561745", "lastRunDuration": "42.958974838256836", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751164561814", "lastRunDuration": "8076.171875", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751164569942", "lastRunDuration": "66.44439697265625", "message": "Memory Usage within limits with current at 50.8791%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751164570009", "lastRunDuration": "3256.709575653076", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751164573323", "lastRunDuration": "49.95608329772949", "message": "Sufficient free space is 2571.26953125 MB"}, "lastRun": {"start": "2025-06-29 02:36:01.595284", "end": "2025-06-29 02:36:13.373806", "duration": "11.778521299362183"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164201601", "lastRunDuration": "116.95194244384766", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164201718", "lastRunDuration": "119.9336051940918", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164202095", "lastRunDuration": "86.74001693725586", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164202298", "lastRunDuration": "45.577287673950195", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:30:01.598085", "end": "2025-06-29 02:30:02.400159", "duration": "0.8020739555358887"}}} 2025-06-29 04:36:45,880 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "13.609886169433594", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "25.968074798583984", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "38.50078582763672", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751164561598", "lastRunDuration": "50.77338218688965", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751164561745", "lastRunDuration": "42.958974838256836", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751164561814", "lastRunDuration": "8076.171875", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751164569942", "lastRunDuration": "66.44439697265625", "message": "Memory Usage within limits with current at 50.8791%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751164570009", "lastRunDuration": "3256.709575653076", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751164573323", "lastRunDuration": "49.95608329772949", "message": "Sufficient free space is 2571.26953125 MB"}, "lastRun": {"start": "2025-06-29 02:36:01.595284", "end": "2025-06-29 02:36:13.373806", "duration": "11.778521299362183"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164201601", "lastRunDuration": "116.95194244384766", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164201718", "lastRunDuration": "119.9336051940918", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164202095", "lastRunDuration": "86.74001693725586", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164202298", "lastRunDuration": "45.577287673950195", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:30:01.598085", "end": "2025-06-29 02:30:02.400159", "duration": "0.8020739555358887"}}} 2025-06-29 04:36:45,882 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found check types executed on VR basic, start: 2025-06-29 02:36:01.595284, end: 2025-06-29 02:36:13.373806, duration: 11.778521299362183 2025-06-29 04:36:45,891 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@8285345- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 04:36:01 CEST 2025, details: service is running which took running duration (ms) 13.609886169433594 2025-06-29 04:36:45,904 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@66570255- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 04:36:01 CEST 2025, details: service is running which took running duration (ms) 25.968074798583984 2025-06-29 04:36:45,915 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@708ce9a0- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 04:36:01 CEST 2025, details: service is running which took running duration (ms) 38.50078582763672 2025-06-29 04:36:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:36:45,926 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2b540cff- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 04:36:01 CEST 2025, details: service is running which took running duration (ms) 50.77338218688965 2025-06-29 04:36:45,937 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@612cd0ab- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 04:36:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 42.958974838256836 2025-06-29 04:36:45,947 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@3f373371- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 04:36:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8076.171875 2025-06-29 04:36:45,959 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@28e11cde- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 04:36:09 CEST 2025, details: Memory Usage within limits with current at 50.8791% which took running duration (ms) 66.44439697265625 2025-06-29 04:36:45,970 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@23d9dec9- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 04:36:10 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3256.709575653076 2025-06-29 04:36:45,982 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2477fb91- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 04:36:13 CEST 2025, details: Sufficient free space is 2571.26953125 MB which took running duration (ms) 49.95608329772949 2025-06-29 04:36:45,982 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found check types executed on VR advanced, start: 2025-06-29 02:30:01.598085, end: 2025-06-29 02:30:02.400159, duration: 0.8020739555358887 2025-06-29 04:36:45,996 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@225de8d5- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 04:30:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 116.95194244384766 2025-06-29 04:36:46,008 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@27a17ac- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 04:30:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 119.9336051940918 2025-06-29 04:36:46,019 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6fb8bebc- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 04:30:02 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 86.74001693725586 2025-06-29 04:36:46,029 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-eed5f28b]) (logid:c16e5b2b) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@74886ba9- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 04:30:02 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 45.577287673950195 2025-06-29 04:36:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-653742e7]) (logid:e6e87c16) No inactive management server node found 2025-06-29 04:36:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-653742e7]) (logid:e6e87c16) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:46,612 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-fa0b8267]) (logid:164b8ef1) HA health check task is running... 2025-06-29 04:36:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:36:46,643 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:36:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:36:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:36:47,067 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:36:47,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:36:47,070 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:36:47,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:36:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e76c1dcc]) (logid:46a94f4e) No inactive management server node found 2025-06-29 04:36:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e76c1dcc]) (logid:46a94f4e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:48,055 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-5:[ctx-771a307a]) (logid:216f9ff1) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:36:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0cda8246]) (logid:9bc0713f) Resetting hosts suitable for reconnect 2025-06-29 04:36:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0cda8246]) (logid:9bc0713f) Completed resetting hosts suitable for reconnect 2025-06-29 04:36:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0cda8246]) (logid:9bc0713f) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:36:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0cda8246]) (logid:9bc0713f) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:36:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0cda8246]) (logid:9bc0713f) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:36:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-0cda8246]) (logid:9bc0713f) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:36:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6d4f3125]) (logid:69db4eb5) VM scheduler.poll is being called at 2025-06-29T02:37:00+0000 2025-06-29 04:36:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6d4f3125]) (logid:69db4eb5) Got 0 scheduled jobs to be executed at 2025-06-29T02:37:00+0000 2025-06-29 04:36:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6d4f3125]) (logid:69db4eb5) Cleaned up 0 VM scheduled job entries 2025-06-29 04:36:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d3cd807d]) (logid:110d3b72) VM Operation Thread Running 2025-06-29 04:36:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1df56814]) (logid:bd9de816) ClusterDRS.poll is being called at 2025-06-29T02:37:00+0000 2025-06-29 04:36:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-1df56814]) (logid:bd9de816) Removed 0 old drs migration plans 2025-06-29 04:36:49,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c40307a]) (logid:7f5fd7cf) No inactive management server node found 2025-06-29 04:36:49,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c40307a]) (logid:7f5fd7cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:50,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8688: Seq 7-8688: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:36:50,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8688 2025-06-29 04:36:50,613 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8fcdeb8f]) (logid:0649e72a) HA health check task is running... 2025-06-29 04:36:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b4c6dc]) (logid:68340f68) No inactive management server node found 2025-06-29 04:36:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b4c6dc]) (logid:68340f68) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f661f343]) (logid:f1760751) No inactive management server node found 2025-06-29 04:36:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f661f343]) (logid:f1760751) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:53,152 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-874eece1]) (logid:f3643b99) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:36:53,163 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-874eece1]) (logid:f3643b99) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:36:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-220aef32]) (logid:c988af9c) No inactive management server node found 2025-06-29 04:36:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-220aef32]) (logid:c988af9c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:54,615 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-caf66aae]) (logid:de209287) HA health check task is running... 2025-06-29 04:36:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b487c49]) (logid:6286b0d4) No inactive management server node found 2025-06-29 04:36:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b487c49]) (logid:6286b0d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b55dd8]) (logid:915df57a) No inactive management server node found 2025-06-29 04:36:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b2b55dd8]) (logid:915df57a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:58,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8d5fe67e]) (logid:fedd2caa) Enabled non-edge zones available for scan: 1 2025-06-29 04:36:58,100 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8d5fe67e]) (logid:fedd2caa) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:36:58,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:36:58,102 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8d5fe67e]) (logid:fedd2caa) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:36:58,103 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Seq 2-4725683383995203838: Routed from 90520738888109 2025-06-29 04:36:58,146 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Seq 1-4725683383995203838: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:36:58,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 04:36:58,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:36:58,176 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Seq 2-3926857400090362137: Routed from 90520738888109 2025-06-29 04:36:58,220 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Seq 2-3926857400090362137: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:36:58,221 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-20fe715e]) (logid:99f02f2c) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 04:36:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a84c5c12]) (logid:c04f4711) Enabled non-edge zones available for scan: 1 2025-06-29 04:36:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a84c5c12]) (logid:c04f4711) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:36:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68aa2780]) (logid:0fe2d3fd) No inactive management server node found 2025-06-29 04:36:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68aa2780]) (logid:0fe2d3fd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:36:58,616 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-85b5eff9]) (logid:e52f109e) HA health check task is running... 2025-06-29 04:36:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e0b7e4c]) (logid:f334f435) No inactive management server node found 2025-06-29 04:36:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5e0b7e4c]) (logid:f334f435) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:00,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8689: Seq 7-8689: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:00,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8689 2025-06-29 04:37:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7160e8b4]) (logid:9495042d) No inactive management server node found 2025-06-29 04:37:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7160e8b4]) (logid:9495042d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:02,617 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9d9402cf]) (logid:c6c39b58) HA health check task is running... 2025-06-29 04:37:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20bc6539]) (logid:81658498) No inactive management server node found 2025-06-29 04:37:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-20bc6539]) (logid:81658498) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bf277fb]) (logid:6e821630) No inactive management server node found 2025-06-29 04:37:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bf277fb]) (logid:6e821630) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2382854b]) (logid:614bede8) No inactive management server node found 2025-06-29 04:37:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2382854b]) (logid:614bede8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:06,619 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-eafe89cb]) (logid:0f7efdf7) HA health check task is running... 2025-06-29 04:37:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a52e1df6]) (logid:60ff6e1f) No inactive management server node found 2025-06-29 04:37:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a52e1df6]) (logid:60ff6e1f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57522b47]) (logid:8437f042) No inactive management server node found 2025-06-29 04:37:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57522b47]) (logid:8437f042) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f97443a2]) (logid:86b4d322) No inactive management server node found 2025-06-29 04:37:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f97443a2]) (logid:86b4d322) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:10,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8690: Seq 7-8690: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:10,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8690 2025-06-29 04:37:10,620 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c635051b]) (logid:f4d30d5e) HA health check task is running... 2025-06-29 04:37:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dc9c8a9]) (logid:85881feb) No inactive management server node found 2025-06-29 04:37:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dc9c8a9]) (logid:85881feb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-0801d586]) (logid:5cb82f9a) Checking if any host reservation can be released ... 2025-06-29 04:37:13,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-0801d586]) (logid:5cb82f9a) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:13,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-0801d586]) (logid:5cb82f9a) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-0801d586]) (logid:5cb82f9a) Done running HostReservationReleaseChecker ... 2025-06-29 04:37:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-60d0558e]) (logid:dd740789) Found 0 routers to update status. 2025-06-29 04:37:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-60d0558e]) (logid:dd740789) Found 0 VPC's to update Redundant State. 2025-06-29 04:37:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-60d0558e]) (logid:dd740789) Found 0 networks to update RvR status. 2025-06-29 04:37:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Running Capacity Checker ... 2025-06-29 04:37:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) recalculating system capacity 2025-06-29 04:37:13,165 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Executing cpu/ram capacity update 2025-06-29 04:37:13,171 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:13,173 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:37:13,190 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:37:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:37:13,207 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:13,208 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:13,214 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:37:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:37:13,216 DEBUG [c.c.c.CapacityManagerImpl] (pool-36-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:37:13,225 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Done executing cpu/ram capacity update 2025-06-29 04:37:13,225 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Executing storage capacity update 2025-06-29 04:37:13,238 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:37:13,239 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:37:13,239 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:37:13,243 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:37:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:37:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:37:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:37:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:37:13,260 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:37:13,261 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:37:13,261 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:37:13,266 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:37:13,272 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:37:13,273 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:37:13,273 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:37:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-37-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:37:13,277 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Done executing storage capacity update 2025-06-29 04:37:13,277 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Executing capacity updates for public ip and Vlans 2025-06-29 04:37:13,302 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Done capacity updates for public ip and Vlans 2025-06-29 04:37:13,302 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Executing capacity updates for private ip 2025-06-29 04:37:13,311 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Done executing capacity updates for private ip 2025-06-29 04:37:13,311 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Done recalculating system capacity 2025-06-29 04:37:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cc39f41]) (logid:d0f17371) No inactive management server node found 2025-06-29 04:37:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cc39f41]) (logid:d0f17371) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:13,361 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-38518521]) (logid:4c7f3711) Done running Capacity Checker ... 2025-06-29 04:37:14,621 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ba7f1aba]) (logid:c362fa2a) HA health check task is running... 2025-06-29 04:37:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1e78b5c]) (logid:82527b42) No inactive management server node found 2025-06-29 04:37:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1e78b5c]) (logid:82527b42) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:15,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8691: Seq 7-8691: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:15,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8691 2025-06-29 04:37:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-820d3f30]) (logid:724df8c0) No inactive management server node found 2025-06-29 04:37:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-820d3f30]) (logid:724df8c0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11f96da4]) (logid:2ddc1a20) No inactive management server node found 2025-06-29 04:37:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11f96da4]) (logid:2ddc1a20) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:18,057 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-6d926b4a]) (logid:ca3e7581) AutoScaling Monitor is running... 2025-06-29 04:37:18,129 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-3e299be7]) (logid:0cc94f95) Starting to remove all volume_stats rows older than [Sat Jun 28 16:37:18 CEST 2025]. 2025-06-29 04:37:18,130 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-3e299be7]) (logid:0cc94f95) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:37:18 CEST 2025]. 2025-06-29 04:37:18,623 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-785396c7]) (logid:d96b57bd) HA health check task is running... 2025-06-29 04:37:18,902 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-bcee5b21]) (logid:707e2521) Starting to remove all vm_stats rows older than [Sat Jun 28 16:37:18 CEST 2025]. 2025-06-29 04:37:18,928 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-bcee5b21]) (logid:707e2521) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:37:18 CEST 2025]. 2025-06-29 04:37:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-168f39ea]) (logid:df20e0fc) No inactive management server node found 2025-06-29 04:37:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-168f39ea]) (logid:df20e0fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3732c637]) (logid:5447c637) No inactive management server node found 2025-06-29 04:37:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3732c637]) (logid:5447c637) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:21,006 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:37:21,011 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:37:21,012 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) Seq 2-4725683383995203839: Routed from 90520738888109 2025-06-29 04:37:21,055 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) Seq 1-4725683383995203839: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:37:21,056 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:37:21,057 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) Seq 2-3926857400090362138: Routed from 90520738888109 2025-06-29 04:37:21,101 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-aef37927]) (logid:ae93f788) Seq 2-3926857400090362138: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:37:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9803373]) (logid:1d25dfe1) No inactive management server node found 2025-06-29 04:37:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9803373]) (logid:1d25dfe1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:22,624 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6873bb0e]) (logid:a93f1c31) HA health check task is running... 2025-06-29 04:37:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a2ff812]) (logid:4f2fdbbc) No inactive management server node found 2025-06-29 04:37:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a2ff812]) (logid:4f2fdbbc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a057ed24]) (logid:56d0ec9a) No inactive management server node found 2025-06-29 04:37:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a057ed24]) (logid:56d0ec9a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:25,368 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8692: Seq 7-8692: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:25,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8692 2025-06-29 04:37:26,025 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:37:26,030 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:37:26,031 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) Seq 2-4725683383995203840: Routed from 90520738888109 2025-06-29 04:37:26,086 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) Seq 1-4725683383995203840: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:37:26,120 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:37:26,121 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) Seq 2-3926857400090362139: Routed from 90520738888109 2025-06-29 04:37:26,225 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-918105b5]) (logid:f368ccf8) Seq 2-3926857400090362139: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:37:26,625 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-acee3d19]) (logid:eb377265) HA health check task is running... 2025-06-29 04:37:26,865 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-248d8cbc]) (logid:82ba6aee) No inactive management server node found 2025-06-29 04:37:26,865 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-248d8cbc]) (logid:82ba6aee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:27,375 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) StorageCollector is running... 2025-06-29 04:37:27,389 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:37:27,389 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:37:27,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:37:27,390 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-3938116399158788164: Routed from 90520738888109 2025-06-29 04:37:27,444 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 4-3938116399158788164: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:37:27,450 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:37:27,450 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:37:27,451 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:37:27,451 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-4725683383995203841: Routed from 90520738888109 2025-06-29 04:37:27,505 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 1-4725683383995203841: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:37:27,517 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:37:27,517 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:37:27,517 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:37:27,519 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-3926857400090362140: Routed from 90520738888109 2025-06-29 04:37:27,534 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-3926857400090362140: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:37:27,546 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:37:27,546 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:37:27,546 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:37:27,547 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-4725683383995203842: Routed from 90520738888109 2025-06-29 04:37:27,590 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 1-4725683383995203842: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:37:27,592 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:37:27,592 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:37:27,593 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:37:27,593 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-3926857400090362141: Routed from 90520738888109 2025-06-29 04:37:27,638 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-7555bead]) (logid:c12bdb3d) Seq 2-3926857400090362141: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:37:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5985337a]) (logid:d0188e42) Enabled non-edge zones available for scan: 1 2025-06-29 04:37:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5985337a]) (logid:d0188e42) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:37:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5985337a]) (logid:d0188e42) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:37:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-30356114]) (logid:057b2c08) Enabled non-edge zones available for scan: 1 2025-06-29 04:37:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-30356114]) (logid:057b2c08) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:37:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac66d986]) (logid:e0695825) No inactive management server node found 2025-06-29 04:37:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac66d986]) (logid:e0695825) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:29,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e0beed1]) (logid:d60b7385) No inactive management server node found 2025-06-29 04:37:29,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e0beed1]) (logid:d60b7385) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:30,626 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8e63b939]) (logid:8725cc3a) HA health check task is running... 2025-06-29 04:37:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bca756b4]) (logid:b8210820) No inactive management server node found 2025-06-29 04:37:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bca756b4]) (logid:b8210820) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63d93324]) (logid:c453cffd) No inactive management server node found 2025-06-29 04:37:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-63d93324]) (logid:c453cffd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8968119]) (logid:bd9acc97) No inactive management server node found 2025-06-29 04:37:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8968119]) (logid:bd9acc97) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:34,628 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e361f5f4]) (logid:c1c0d451) HA health check task is running... 2025-06-29 04:37:34,808 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 1-4725683383995203585: Processing: { Ans: , MgmtId: 90520738888109, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:35,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8693: Seq 7-8693: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:35,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8693 2025-06-29 04:37:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ee62314]) (logid:0259032e) No inactive management server node found 2025-06-29 04:37:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ee62314]) (logid:0259032e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5ef7181]) (logid:180f4823) No inactive management server node found 2025-06-29 04:37:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5ef7181]) (logid:180f4823) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:38,629 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f29be7dc]) (logid:44756b7c) HA health check task is running... 2025-06-29 04:37:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b140b589]) (logid:9413c3cf) No inactive management server node found 2025-06-29 04:37:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b140b589]) (logid:9413c3cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aea70530]) (logid:a5ee28a7) No inactive management server node found 2025-06-29 04:37:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aea70530]) (logid:a5ee28a7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f4ca0dc]) (logid:bb00ed00) No inactive management server node found 2025-06-29 04:37:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f4ca0dc]) (logid:bb00ed00) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:42,630 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-e267874f]) (logid:2501e04a) HA health check task is running... 2025-06-29 04:37:43,005 DEBUG [o.a.c.s.o.B.BucketUsageTask] (Bucket-Usage-1:[ctx-1b2a9282]) (logid:27b16e8f) Completed updating bucket usage for all object stores 2025-06-29 04:37:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-f3c87884]) (logid:b6ec908d) DbCollector is running... 2025-06-29 04:37:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fcd8d927]) (logid:783f2267) Found 0 routers to update status. 2025-06-29 04:37:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fcd8d927]) (logid:783f2267) Found 0 VPC's to update Redundant State. 2025-06-29 04:37:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-fcd8d927]) (logid:783f2267) Found 0 networks to update RvR status. 2025-06-29 04:37:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bec70fa]) (logid:46e2a245) No inactive management server node found 2025-06-29 04:37:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bec70fa]) (logid:46e2a245) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) ManagementServerCollector is running... 2025-06-29 04:37:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) DB properties were already loaded 2025-06-29 04:37:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:37:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Execution of process [4067464] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:37:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Exit value of process [4067464] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:37:44,058 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Process [4067464] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:37:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) usage install: null 2025-06-29 04:37:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Sessions active: 0 2025-06-29 04:37:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:37:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067467] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:37:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:37:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067472] for command [/bin/bash -c uname -r ]. 2025-06-29 04:37:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:37:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067473] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:37:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) system memory from /proc: 4887515136 2025-06-29 04:37:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:37:44,088 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067478] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:37:44,088 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) free memory from /proc: 326311936 2025-06-29 04:37:44,088 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:37:44,106 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067483] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:37:44,106 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) used memory from /proc: 1019588 2025-06-29 04:37:44,106 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:37:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067484] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:37:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:37:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067485] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:37:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:37:44,132 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067486] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:37:44,132 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:37:44,137 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067493] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:37:44,137 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:37:44,143 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Successfully executed process [4067494] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail buffersdirect.capacity, 326696 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail buffersdirect.count, 96 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail buffersdirect.used, 326696 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail buffersmapped.capacity, 0 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail buffersmapped.count, 0 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail buffersmapped.used, 0 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:37:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail jvmuptime, 3723850 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memoryheap.committed, 654311424 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memoryheap.init, 77594624 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memoryheap.usage, 0.13167475422998276 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorynon-heap.committed, 198574080 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorynon-heap.max, -1 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorynon-heap.usage, -1.94065736E8 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorynon-heap.used, 194065736 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11447364036256998 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21432707368280735 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388065457344055 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.Metaspace.usage, 0.9927110269036091 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.PS-Eden-Space.usage, 0.0895639522530973 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorytotal.committed, 852885504 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorytotal.init, 85262336 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorytotal.max, 1908932607 2025-06-29 04:37:44,147 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail memorytotal.used, 445423968 2025-06-29 04:37:44,148 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail threadsdeadlocks, [] 2025-06-29 04:37:44,148 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail threadsnew.count, 0 2025-06-29 04:37:44,149 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) not storing detail threadstimed_waiting.count, 82 2025-06-29 04:37:44,151 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-94eeca35]) (logid:90560f80) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":403850128,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3723762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":444523832,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":251358232,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":326311936,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.04,0.14],"systemCyclesUsage":[66419223,53838948,4590037720],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:37:44,151 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 123, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":403850128,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3723762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":444523832,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":251358232,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":326311936,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.04,0.14],"systemCyclesUsage":[66419223,53838948,4590037720],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:37:44,151 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:37:44,158 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:37:44,158 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 123, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":403850128,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3723762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":444523832,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":251358232,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":326311936,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.04,0.14],"systemCyclesUsage":[66419223,53838948,4590037720],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:37:44,159 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-27:[ctx-8bfb158e]) (logid:16cfc2c5) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:37:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":403850128,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3723762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.03,"totalInit":85262336,"totalUsed":444523832,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":251358232,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":201,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":326311936,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.03,0.04,0.14],"systemCyclesUsage":[66419223,53838948,4590037720],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:37:44,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7b39b8d]) (logid:e48e1f95) No inactive management server node found 2025-06-29 04:37:44,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7b39b8d]) (logid:e48e1f95) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:45,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8694: Seq 7-8694: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:45,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8694 2025-06-29 04:37:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:37:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:37:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16a58310]) (logid:69f54eac) No inactive management server node found 2025-06-29 04:37:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16a58310]) (logid:69f54eac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:46,631 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-b7653daf]) (logid:6f2c8a8d) HA health check task is running... 2025-06-29 04:37:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:37:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:37:47,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:47,087 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:37:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:37:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-107c51e9]) (logid:817b8f96) No inactive management server node found 2025-06-29 04:37:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-107c51e9]) (logid:817b8f96) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b04672f4]) (logid:dae93d3c) VM scheduler.poll is being called at 2025-06-29T02:38:00+0000 2025-06-29 04:37:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b04672f4]) (logid:dae93d3c) Got 0 scheduled jobs to be executed at 2025-06-29T02:38:00+0000 2025-06-29 04:37:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b04672f4]) (logid:dae93d3c) Cleaned up 0 VM scheduled job entries 2025-06-29 04:37:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-52ff11c3]) (logid:69d4ab5e) VM Operation Thread Running 2025-06-29 04:37:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-104a5cbe]) (logid:74d73810) ClusterDRS.poll is being called at 2025-06-29T02:38:00+0000 2025-06-29 04:37:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-104a5cbe]) (logid:74d73810) Removed 0 old drs migration plans 2025-06-29 04:37:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f7885ba]) (logid:509c71de) No inactive management server node found 2025-06-29 04:37:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8f7885ba]) (logid:509c71de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:50,632 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-80b8c0cf]) (logid:0bf12818) HA health check task is running... 2025-06-29 04:37:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee1e78a8]) (logid:88b4852e) No inactive management server node found 2025-06-29 04:37:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee1e78a8]) (logid:88b4852e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed6e2aea]) (logid:fcf721c8) No inactive management server node found 2025-06-29 04:37:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed6e2aea]) (logid:fcf721c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0897c4ce]) (logid:7bc68232) No inactive management server node found 2025-06-29 04:37:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0897c4ce]) (logid:7bc68232) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:54,634 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-1ebda7a4]) (logid:f384a36f) HA health check task is running... 2025-06-29 04:37:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99508d2e]) (logid:2df396a1) No inactive management server node found 2025-06-29 04:37:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99508d2e]) (logid:2df396a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:55,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8696: Seq 7-8696: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:37:55,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8696 2025-06-29 04:37:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f568d91]) (logid:0010accd) No inactive management server node found 2025-06-29 04:37:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f568d91]) (logid:0010accd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:58,091 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3954fca8]) (logid:c5f7e04d) Enabled non-edge zones available for scan: 1 2025-06-29 04:37:58,098 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-3954fca8]) (logid:c5f7e04d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:37:58,099 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-3954fca8]) (logid:c5f7e04d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:37:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c1ae718b]) (logid:8a0a8bf7) Enabled non-edge zones available for scan: 1 2025-06-29 04:37:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c1ae718b]) (logid:8a0a8bf7) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:37:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f52b5266]) (logid:7920c010) No inactive management server node found 2025-06-29 04:37:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f52b5266]) (logid:7920c010) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:37:58,635 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3d99c524]) (logid:973b0a91) HA health check task is running... 2025-06-29 04:37:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bdef376]) (logid:a8d5b6ed) No inactive management server node found 2025-06-29 04:37:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3bdef376]) (logid:a8d5b6ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:01,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94dee833]) (logid:cec5aad7) No inactive management server node found 2025-06-29 04:38:01,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94dee833]) (logid:cec5aad7) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:02,636 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2b36149a]) (logid:bfbb79eb) HA health check task is running... 2025-06-29 04:38:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d756388]) (logid:2a09ef83) No inactive management server node found 2025-06-29 04:38:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d756388]) (logid:2a09ef83) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:04,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-600e8c5b]) (logid:6d43a593) No inactive management server node found 2025-06-29 04:38:04,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-600e8c5b]) (logid:6d43a593) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:05,372 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8697: Seq 7-8697: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:38:05,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8697 2025-06-29 04:38:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4635630]) (logid:8c040626) No inactive management server node found 2025-06-29 04:38:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4635630]) (logid:8c040626) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:38:06,638 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7f19739b]) (logid:3f622dd0) HA health check task is running... 2025-06-29 04:38:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc25053c]) (logid:812b7920) No inactive management server node found 2025-06-29 04:38:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc25053c]) (logid:812b7920) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee1c62d1]) (logid:69ca9ab0) No inactive management server node found 2025-06-29 04:38:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee1c62d1]) (logid:69ca9ab0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e3c8bec]) (logid:777961ee) No inactive management server node found 2025-06-29 04:38:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e3c8bec]) (logid:777961ee) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:10,639 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c0858ba1]) (logid:62ed6330) HA health check task is running... 2025-06-29 04:38:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-995634a3]) (logid:784ba59c) No inactive management server node found 2025-06-29 04:38:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-995634a3]) (logid:784ba59c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5adba8d2]) (logid:f9c03aac) Found 0 routers to update status. 2025-06-29 04:38:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5adba8d2]) (logid:f9c03aac) Found 0 VPC's to update Redundant State. 2025-06-29 04:38:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-5adba8d2]) (logid:f9c03aac) Found 0 networks to update RvR status. 2025-06-29 04:38:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27bba2bf]) (logid:f828b077) No inactive management server node found 2025-06-29 04:38:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27bba2bf]) (logid:f828b077) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:14,640 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-18bf2435]) (logid:6c154607) HA health check task is running... 2025-06-29 04:38:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-499a7d21]) (logid:fa27394b) No inactive management server node found 2025-06-29 04:38:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-499a7d21]) (logid:fa27394b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:15,370 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8698: Seq 7-8698: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:38:15,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8698 2025-06-29 04:38:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a4787d2]) (logid:09445165) No inactive management server node found 2025-06-29 04:38:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a4787d2]) (logid:09445165) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:17,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4faf3728]) (logid:8aa99b4f) No inactive management server node found 2025-06-29 04:38:17,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4faf3728]) (logid:8aa99b4f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:18,061 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-202783d1]) (logid:374a0de9) AutoScaling Monitor is running... 2025-06-29 04:38:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1216a84e]) (logid:997f8f85) Resetting hosts suitable for reconnect 2025-06-29 04:38:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1216a84e]) (logid:997f8f85) Completed resetting hosts suitable for reconnect 2025-06-29 04:38:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1216a84e]) (logid:997f8f85) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:38:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1216a84e]) (logid:997f8f85) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:38:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1216a84e]) (logid:997f8f85) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:38:18,099 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-1216a84e]) (logid:997f8f85) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:38:18,132 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-aed19900]) (logid:f2b28aa5) Starting to remove all volume_stats rows older than [Sat Jun 28 16:38:18 CEST 2025]. 2025-06-29 04:38:18,133 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-aed19900]) (logid:f2b28aa5) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:38:18 CEST 2025]. 2025-06-29 04:38:18,642 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-a3fe9d56]) (logid:5f129285) HA health check task is running... 2025-06-29 04:38:18,929 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-9f27000b]) (logid:01b8028e) Starting to remove all vm_stats rows older than [Sat Jun 28 16:38:18 CEST 2025]. 2025-06-29 04:38:18,953 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-9f27000b]) (logid:01b8028e) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:38:18 CEST 2025]. 2025-06-29 04:38:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-362663cf]) (logid:b52c9f15) No inactive management server node found 2025-06-29 04:38:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-362663cf]) (logid:b52c9f15) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:20,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8699: Seq 7-8699: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:38:20,386 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8699 2025-06-29 04:38:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-317c1060]) (logid:d8d81c83) No inactive management server node found 2025-06-29 04:38:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-317c1060]) (logid:d8d81c83) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:21,104 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:38:21,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:38:21,106 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) Seq 2-4725683383995203843: Routed from 90520738888109 2025-06-29 04:38:21,150 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) Seq 1-4725683383995203843: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:38:21,151 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:38:21,152 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) Seq 2-3926857400090362142: Routed from 90520738888109 2025-06-29 04:38:21,196 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-d54cb61c]) (logid:88c3889f) Seq 2-3926857400090362142: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:38:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d84a772]) (logid:8841f1f9) No inactive management server node found 2025-06-29 04:38:22,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d84a772]) (logid:8841f1f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:22,643 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-20837192]) (logid:ebca8972) HA health check task is running... 2025-06-29 04:38:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1715a4ca]) (logid:3fc91700) No inactive management server node found 2025-06-29 04:38:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1715a4ca]) (logid:3fc91700) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27307d53]) (logid:f217877f) No inactive management server node found 2025-06-29 04:38:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27307d53]) (logid:f217877f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:26,279 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:38:26,284 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:38:26,285 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) Seq 2-4725683383995203844: Routed from 90520738888109 2025-06-29 04:38:26,342 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) Seq 1-4725683383995203844: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:38:26,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:38:26,379 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) Seq 2-3926857400090362143: Routed from 90520738888109 2025-06-29 04:38:26,475 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b21a1f96]) (logid:4b71cb0c) Seq 2-3926857400090362143: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:38:26,644 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-aad4cc1a]) (logid:e404b7b5) HA health check task is running... 2025-06-29 04:38:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-402653a0]) (logid:c62190ed) No inactive management server node found 2025-06-29 04:38:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-402653a0]) (logid:c62190ed) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:27,638 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) StorageCollector is running... 2025-06-29 04:38:27,644 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:38:27,644 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:38:27,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:38:27,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3938116399158788165: Routed from 90520738888109 2025-06-29 04:38:27,698 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 4-3938116399158788165: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:38:27,702 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:38:27,702 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:38:27,702 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:38:27,703 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3926857400090362144: Routed from 90520738888109 2025-06-29 04:38:27,715 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3926857400090362144: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:38:27,726 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:38:27,726 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:38:27,726 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:38:27,727 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3926857400090362145: Routed from 90520738888109 2025-06-29 04:38:27,818 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3926857400090362145: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:38:27,828 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:38:27,828 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:38:27,828 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:38:27,829 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-4725683383995203845: Routed from 90520738888109 2025-06-29 04:38:27,873 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 1-4725683383995203845: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:38:27,876 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:38:27,876 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:38:27,876 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:38:27,877 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3926857400090362146: Routed from 90520738888109 2025-06-29 04:38:27,921 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-bd933608]) (logid:81e8612f) Seq 2-3926857400090362146: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:38:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d0d2b826]) (logid:815378e2) Enabled non-edge zones available for scan: 1 2025-06-29 04:38:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d0d2b826]) (logid:815378e2) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:38:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d0d2b826]) (logid:815378e2) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:38:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-47bee302]) (logid:fbdd6c47) Enabled non-edge zones available for scan: 1 2025-06-29 04:38:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-47bee302]) (logid:fbdd6c47) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:38:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d301fd3]) (logid:36bfc0be) No inactive management server node found 2025-06-29 04:38:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d301fd3]) (logid:36bfc0be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-860ee87a]) (logid:d20c3dba) No inactive management server node found 2025-06-29 04:38:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-860ee87a]) (logid:d20c3dba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:30,371 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8700: Seq 7-8700: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:38:30,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8700 2025-06-29 04:38:30,645 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5b154422]) (logid:a9e65412) HA health check task is running... 2025-06-29 04:38:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75f86027]) (logid:5df27ced) No inactive management server node found 2025-06-29 04:38:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75f86027]) (logid:5df27ced) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16018814]) (logid:b0a79b25) No inactive management server node found 2025-06-29 04:38:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16018814]) (logid:b0a79b25) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e16ee486]) (logid:37835527) No inactive management server node found 2025-06-29 04:38:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e16ee486]) (logid:37835527) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:34,647 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-87da5ef7]) (logid:6f4d6484) HA health check task is running... 2025-06-29 04:38:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a800311]) (logid:9bca4f4b) No inactive management server node found 2025-06-29 04:38:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3a800311]) (logid:9bca4f4b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fc48313]) (logid:0ac31600) No inactive management server node found 2025-06-29 04:38:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9fc48313]) (logid:0ac31600) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:38,648 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-151714bf]) (logid:fca9e6a8) HA health check task is running... 2025-06-29 04:38:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-627c1718]) (logid:a1a86463) No inactive management server node found 2025-06-29 04:38:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-627c1718]) (logid:a1a86463) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb098478]) (logid:bc4e0863) No inactive management server node found 2025-06-29 04:38:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb098478]) (logid:bc4e0863) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:40,371 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8701: Seq 7-8701: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:38:40,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8701 2025-06-29 04:38:41,863 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-089c7e1c]) (logid:0a7555af) No inactive management server node found 2025-06-29 04:38:41,863 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-089c7e1c]) (logid:0a7555af) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:42,649 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4e463600]) (logid:a2eff610) HA health check task is running... 2025-06-29 04:38:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-887c545d]) (logid:33442bf8) DbCollector is running... 2025-06-29 04:38:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-827cd6ad]) (logid:62542e4e) Found 0 routers to update status. 2025-06-29 04:38:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-827cd6ad]) (logid:62542e4e) Found 0 VPC's to update Redundant State. 2025-06-29 04:38:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-827cd6ad]) (logid:62542e4e) Found 0 networks to update RvR status. 2025-06-29 04:38:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25881154]) (logid:176f38cf) No inactive management server node found 2025-06-29 04:38:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-25881154]) (logid:176f38cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) ManagementServerCollector is running... 2025-06-29 04:38:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) DB properties were already loaded 2025-06-29 04:38:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:38:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Execution of process [4067507] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:38:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Exit value of process [4067507] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:38:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Process [4067507] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:38:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) usage install: null 2025-06-29 04:38:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Sessions active: 0 2025-06-29 04:38:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:38:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067510] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:38:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:38:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067515] for command [/bin/bash -c uname -r ]. 2025-06-29 04:38:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:38:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067516] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:38:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) system memory from /proc: 4887515136 2025-06-29 04:38:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:38:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067521] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:38:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) free memory from /proc: 344834048 2025-06-29 04:38:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:38:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067526] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:38:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) used memory from /proc: 1019588 2025-06-29 04:38:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:38:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067527] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:38:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:38:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067528] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:38:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:38:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067529] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:38:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:38:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067536] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:38:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:38:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Successfully executed process [4067537] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail buffersdirect.capacity, 326696 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail buffersdirect.count, 96 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail buffersdirect.used, 326696 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail buffersmapped.capacity, 0 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail buffersmapped.count, 0 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail buffersmapped.used, 0 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail jvmuptime, 3783832 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:38:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memoryheap.committed, 654311424 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memoryheap.init, 77594624 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memoryheap.usage, 0.1439605603929209 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorynon-heap.committed, 198574080 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorynon-heap.max, -1 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorynon-heap.usage, -1.94186456E8 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorynon-heap.used, 194186456 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11512138429752067 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21461658946245876 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388065457344055 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.Metaspace.usage, 0.9927505708076585 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.PS-Eden-Space.usage, 0.12560957694226557 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorytotal.committed, 852885504 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorytotal.init, 85262336 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorytotal.max, 1908932607 2025-06-29 04:38:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail memorytotal.used, 468997464 2025-06-29 04:38:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail threadsdeadlocks, [] 2025-06-29 04:38:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail threadsnew.count, 0 2025-06-29 04:38:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:38:44,132 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-f974ff96]) (logid:e142780e) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":380261608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3783759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":468235376,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274811008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":344834048,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.13],"systemCyclesUsage":[66419548,53839253,4590060842],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:38:44,133 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 125, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":380261608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3783759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":468235376,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274811008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":344834048,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.13],"systemCyclesUsage":[66419548,53839253,4590060842],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:38:44,133 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:38:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:38:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 125, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":380261608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3783759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":468235376,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274811008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":344834048,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.13],"systemCyclesUsage":[66419548,53839253,4590060842],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:38:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-27:[ctx-2cfacc09]) (logid:c12ca084) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:38:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":380261608,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3783759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.01,"totalInit":85262336,"totalUsed":468235376,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274811008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":344834048,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.01,0.03,0.13],"systemCyclesUsage":[66419548,53839253,4590060842],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:38:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c35df7e6]) (logid:b62a497f) No inactive management server node found 2025-06-29 04:38:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c35df7e6]) (logid:b62a497f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:38:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:38:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e8adab3]) (logid:c413f043) No inactive management server node found 2025-06-29 04:38:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e8adab3]) (logid:c413f043) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:38:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:38:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:38:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:38:46,651 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e78c194c]) (logid:6a938bbf) HA health check task is running... 2025-06-29 04:38:47,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:38:47,077 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:38:47,078 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:38:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:38:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d516a30]) (logid:5b2b7aeb) No inactive management server node found 2025-06-29 04:38:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d516a30]) (logid:5b2b7aeb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f293289d]) (logid:6ed756d2) VM scheduler.poll is being called at 2025-06-29T02:39:00+0000 2025-06-29 04:38:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f293289d]) (logid:6ed756d2) Got 0 scheduled jobs to be executed at 2025-06-29T02:39:00+0000 2025-06-29 04:38:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f293289d]) (logid:6ed756d2) Cleaned up 0 VM scheduled job entries 2025-06-29 04:38:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ed7ba181]) (logid:81518ad0) VM Operation Thread Running 2025-06-29 04:38:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e391c2dc]) (logid:206712fc) ClusterDRS.poll is being called at 2025-06-29T02:39:00+0000 2025-06-29 04:38:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e391c2dc]) (logid:206712fc) Removed 0 old drs migration plans 2025-06-29 04:38:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87896780]) (logid:e7885c61) No inactive management server node found 2025-06-29 04:38:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87896780]) (logid:e7885c61) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:50,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8703: Seq 7-8703: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:38:50,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8703 2025-06-29 04:38:50,652 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-dbcb8913]) (logid:5c3e7833) HA health check task is running... 2025-06-29 04:38:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5a51294]) (logid:276c27f8) No inactive management server node found 2025-06-29 04:38:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5a51294]) (logid:276c27f8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83e0bd22]) (logid:b4ce2c14) No inactive management server node found 2025-06-29 04:38:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83e0bd22]) (logid:b4ce2c14) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4346ee4]) (logid:2b5f9fed) No inactive management server node found 2025-06-29 04:38:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4346ee4]) (logid:2b5f9fed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:54,653 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-36593016]) (logid:459f0d56) HA health check task is running... 2025-06-29 04:38:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aeccb288]) (logid:3d4f340e) No inactive management server node found 2025-06-29 04:38:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aeccb288]) (logid:3d4f340e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca935bb9]) (logid:f7661219) No inactive management server node found 2025-06-29 04:38:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca935bb9]) (logid:f7661219) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c105d56]) (logid:b6f0d574) Enabled non-edge zones available for scan: 1 2025-06-29 04:38:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0c105d56]) (logid:b6f0d574) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:38:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0c105d56]) (logid:b6f0d574) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:38:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d47847d5]) (logid:61319eb3) Enabled non-edge zones available for scan: 1 2025-06-29 04:38:58,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d47847d5]) (logid:61319eb3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:38:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34cf8a44]) (logid:3446db1f) No inactive management server node found 2025-06-29 04:38:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34cf8a44]) (logid:3446db1f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:38:58,654 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d258eb27]) (logid:d9e6a910) HA health check task is running... 2025-06-29 04:38:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e99947dd]) (logid:4510dd3b) No inactive management server node found 2025-06-29 04:38:59,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e99947dd]) (logid:4510dd3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:00,372 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8704: Seq 7-8704: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:00,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8704 2025-06-29 04:39:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e18f6a9]) (logid:a5e4cfca) No inactive management server node found 2025-06-29 04:39:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e18f6a9]) (logid:a5e4cfca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:02,656 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f1e3e1d4]) (logid:f9369dbd) HA health check task is running... 2025-06-29 04:39:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4abf768c]) (logid:c7f86a4b) No inactive management server node found 2025-06-29 04:39:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4abf768c]) (logid:c7f86a4b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e150529]) (logid:36d28948) No inactive management server node found 2025-06-29 04:39:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e150529]) (logid:36d28948) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34f11900]) (logid:7b77fc44) No inactive management server node found 2025-06-29 04:39:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34f11900]) (logid:7b77fc44) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:06,657 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f245d982]) (logid:c81e0536) HA health check task is running... 2025-06-29 04:39:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30f4eff0]) (logid:fe1c103e) No inactive management server node found 2025-06-29 04:39:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30f4eff0]) (logid:fe1c103e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1beca945]) (logid:ab8fb730) No inactive management server node found 2025-06-29 04:39:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1beca945]) (logid:ab8fb730) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c53854e]) (logid:1605f76c) No inactive management server node found 2025-06-29 04:39:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0c53854e]) (logid:1605f76c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:10,372 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8705: Seq 7-8705: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:10,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8705 2025-06-29 04:39:10,658 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-396361fb]) (logid:ed9a6a89) HA health check task is running... 2025-06-29 04:39:11,864 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b3cb112]) (logid:0ac307e5) No inactive management server node found 2025-06-29 04:39:11,864 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b3cb112]) (logid:0ac307e5) Peer scan is finished. profiler: Done. Duration: 6ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 4ms 2025-06-29 04:39:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2929285d]) (logid:627b1398) Found 0 routers to update status. 2025-06-29 04:39:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2929285d]) (logid:627b1398) Found 0 VPC's to update Redundant State. 2025-06-29 04:39:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2929285d]) (logid:627b1398) Found 0 networks to update RvR status. 2025-06-29 04:39:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b6ccf43]) (logid:e851ed90) No inactive management server node found 2025-06-29 04:39:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b6ccf43]) (logid:e851ed90) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:14,659 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2519d3a9]) (logid:d6d9c06c) HA health check task is running... 2025-06-29 04:39:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-619e738b]) (logid:2b06c6da) No inactive management server node found 2025-06-29 04:39:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-619e738b]) (logid:2b06c6da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45146a8c]) (logid:2aae4e8d) No inactive management server node found 2025-06-29 04:39:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45146a8c]) (logid:2aae4e8d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:17,867 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db4d6c63]) (logid:4ff836f8) No inactive management server node found 2025-06-29 04:39:17,867 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db4d6c63]) (logid:4ff836f8) Peer scan is finished. profiler: Done. Duration: 8ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 7ms 2025-06-29 04:39:18,063 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-33d9b1fe]) (logid:eb5409c2) AutoScaling Monitor is running... 2025-06-29 04:39:18,135 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-5b4f4ced]) (logid:308139b9) Starting to remove all volume_stats rows older than [Sat Jun 28 16:39:18 CEST 2025]. 2025-06-29 04:39:18,135 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-5b4f4ced]) (logid:308139b9) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:39:18 CEST 2025]. 2025-06-29 04:39:18,661 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7527c36a]) (logid:b16c2aaa) HA health check task is running... 2025-06-29 04:39:18,955 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-471e80ed]) (logid:37b2ed54) Starting to remove all vm_stats rows older than [Sat Jun 28 16:39:18 CEST 2025]. 2025-06-29 04:39:18,978 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-471e80ed]) (logid:37b2ed54) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:39:18 CEST 2025]. 2025-06-29 04:39:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-746179bd]) (logid:47033844) No inactive management server node found 2025-06-29 04:39:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-746179bd]) (logid:47033844) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:20,386 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8706: Seq 7-8706: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:20,395 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8706 2025-06-29 04:39:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87f4144f]) (logid:968cf109) No inactive management server node found 2025-06-29 04:39:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87f4144f]) (logid:968cf109) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:21,198 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:39:21,200 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:39:21,202 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) Seq 2-4725683383995203846: Routed from 90520738888109 2025-06-29 04:39:21,244 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) Seq 1-4725683383995203846: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:39:21,246 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:39:21,247 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) Seq 2-3926857400090362147: Routed from 90520738888109 2025-06-29 04:39:21,290 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-0ccb61d9]) (logid:1e5ef2fe) Seq 2-3926857400090362147: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:39:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db70e465]) (logid:648e4093) No inactive management server node found 2025-06-29 04:39:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db70e465]) (logid:648e4093) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:22,662 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-669611aa]) (logid:182f870f) HA health check task is running... 2025-06-29 04:39:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2ee193]) (logid:1c2db21c) No inactive management server node found 2025-06-29 04:39:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca2ee193]) (logid:1c2db21c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9791658e]) (logid:3ba290b1) No inactive management server node found 2025-06-29 04:39:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9791658e]) (logid:3ba290b1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:26,533 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:39:26,537 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:39:26,538 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) Seq 2-4725683383995203847: Routed from 90520738888109 2025-06-29 04:39:26,591 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) Seq 1-4725683383995203847: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:39:26,625 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:39:26,625 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) Seq 2-3926857400090362148: Routed from 90520738888109 2025-06-29 04:39:26,663 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-16f8b34f]) (logid:16f89ef9) HA health check task is running... 2025-06-29 04:39:26,710 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-e586a25f]) (logid:9e9a06e0) Seq 2-3926857400090362148: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:39:26,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca363e14]) (logid:04f51329) No inactive management server node found 2025-06-29 04:39:26,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca363e14]) (logid:04f51329) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:27,921 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) StorageCollector is running... 2025-06-29 04:39:27,928 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:39:27,928 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:39:27,928 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:39:27,929 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3938116399158788166: Routed from 90520738888109 2025-06-29 04:39:27,983 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 4-3938116399158788166: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:39:27,986 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:39:27,986 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:39:27,986 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:39:27,988 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3926857400090362149: Routed from 90520738888109 2025-06-29 04:39:27,998 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3926857400090362149: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:39:28,010 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:39:28,010 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:39:28,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:39:28,011 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3926857400090362150: Routed from 90520738888109 2025-06-29 04:39:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-41ef7941]) (logid:35a25134) Enabled non-edge zones available for scan: 1 2025-06-29 04:39:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-41ef7941]) (logid:35a25134) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:39:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-41ef7941]) (logid:35a25134) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:39:28,105 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3926857400090362150: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:39:28,115 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:39:28,115 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:39:28,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:39:28,116 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-4725683383995203848: Routed from 90520738888109 2025-06-29 04:39:28,161 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 1-4725683383995203848: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:39:28,163 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:39:28,163 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:39:28,163 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:39:28,164 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3926857400090362151: Routed from 90520738888109 2025-06-29 04:39:28,208 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-3f657faf]) (logid:e44f5f29) Seq 2-3926857400090362151: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:39:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-bd83a54b]) (logid:5bcabe69) Enabled non-edge zones available for scan: 1 2025-06-29 04:39:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-bd83a54b]) (logid:5bcabe69) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:39:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-936a6e75]) (logid:25596d2f) No inactive management server node found 2025-06-29 04:39:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-936a6e75]) (logid:25596d2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:39:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94e2bc59]) (logid:bdb238a2) No inactive management server node found 2025-06-29 04:39:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94e2bc59]) (logid:bdb238a2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:30,373 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8707: Seq 7-8707: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:30,385 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8707 2025-06-29 04:39:30,664 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-86f1c193]) (logid:7adc391d) HA health check task is running... 2025-06-29 04:39:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-465a0897]) (logid:c4be5d5c) No inactive management server node found 2025-06-29 04:39:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-465a0897]) (logid:c4be5d5c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8095b123]) (logid:56b996a9) No inactive management server node found 2025-06-29 04:39:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8095b123]) (logid:56b996a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a71ff22]) (logid:5bc671cf) No inactive management server node found 2025-06-29 04:39:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a71ff22]) (logid:5bc671cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:34,666 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-de7a7e48]) (logid:a2431e9f) HA health check task is running... 2025-06-29 04:39:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-500e7e5d]) (logid:97e99508) No inactive management server node found 2025-06-29 04:39:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-500e7e5d]) (logid:97e99508) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7174e989]) (logid:2b450e57) No inactive management server node found 2025-06-29 04:39:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7174e989]) (logid:2b450e57) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:38,667 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-307a2478]) (logid:a2b052ad) HA health check task is running... 2025-06-29 04:39:38,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43e380d2]) (logid:d03af506) No inactive management server node found 2025-06-29 04:39:38,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43e380d2]) (logid:d03af506) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8eaf0301]) (logid:90796e94) No inactive management server node found 2025-06-29 04:39:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8eaf0301]) (logid:90796e94) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:40,374 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8708: Seq 7-8708: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:40,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8708 2025-06-29 04:39:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a968cda3]) (logid:570e6afd) No inactive management server node found 2025-06-29 04:39:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a968cda3]) (logid:570e6afd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:42,669 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3c7723be]) (logid:ef80adb1) HA health check task is running... 2025-06-29 04:39:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-e1bf49eb]) (logid:0c9449dc) DbCollector is running... 2025-06-29 04:39:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e2b5df9a]) (logid:e1eb082a) Found 0 routers to update status. 2025-06-29 04:39:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e2b5df9a]) (logid:e1eb082a) Found 0 VPC's to update Redundant State. 2025-06-29 04:39:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e2b5df9a]) (logid:e1eb082a) Found 0 networks to update RvR status. 2025-06-29 04:39:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95aea2af]) (logid:cd526c11) No inactive management server node found 2025-06-29 04:39:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95aea2af]) (logid:cd526c11) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) ManagementServerCollector is running... 2025-06-29 04:39:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) DB properties were already loaded 2025-06-29 04:39:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:39:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Execution of process [4067542] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:39:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Exit value of process [4067542] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:39:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Process [4067542] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:39:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) usage install: null 2025-06-29 04:39:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Sessions active: 0 2025-06-29 04:39:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:39:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067545] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:39:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:39:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067550] for command [/bin/bash -c uname -r ]. 2025-06-29 04:39:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:39:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067551] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:39:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) system memory from /proc: 4887515136 2025-06-29 04:39:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:39:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067556] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:39:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) free memory from /proc: 335110144 2025-06-29 04:39:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:39:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067561] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:39:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) used memory from /proc: 1019588 2025-06-29 04:39:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:39:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067562] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:39:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:39:44,103 DEBUG [o.a.c.m.w.W.WebhookDeliveryCleanupWorker] (Webhook-Deliveries-Cleanup-Worker-1:[ctx-aec6dfaf]) (logid:46f128ac) Clearing old deliveries for webhooks with limit=10 using management server ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:39:44,105 DEBUG [o.a.c.m.w.W.WebhookDeliveryCleanupWorker] (Webhook-Deliveries-Cleanup-Worker-1:[ctx-aec6dfaf]) (logid:46f128ac) Cleared old deliveries with limit=10 for 0 webhooks 2025-06-29 04:39:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067563] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:39:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:39:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067564] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:39:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:39:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067571] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:39:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:39:44,129 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Successfully executed process [4067572] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail buffersdirect.capacity, 326696 2025-06-29 04:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail buffersdirect.count, 96 2025-06-29 04:39:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail buffersdirect.used, 326696 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail buffersmapped.capacity, 0 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail buffersmapped.count, 0 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail buffersmapped.used, 0 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail jvmuptime, 3843835 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memoryheap.committed, 654311424 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memoryheap.init, 77594624 2025-06-29 04:39:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memoryheap.usage, 0.15510594704032632 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorynon-heap.committed, 198574080 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorynon-heap.max, -1 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorynon-heap.usage, -1.94307632E8 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorynon-heap.used, 194307632 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11528904792055451 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2151956210217616 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388065457344055 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.Metaspace.usage, 0.9929611850792254 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.PS-Eden-Space.usage, 0.1583092971543551 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorytotal.committed, 852885504 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorytotal.init, 85262336 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorytotal.max, 1908932607 2025-06-29 04:39:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail memorytotal.used, 490395504 2025-06-29 04:39:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail threadsdeadlocks, [] 2025-06-29 04:39:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail threadsnew.count, 0 2025-06-29 04:39:44,136 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:39:44,138 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-628a309a]) (logid:504aecfa) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":359174544,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3843761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":489438944,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":296086800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335110144,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.12],"systemCyclesUsage":[66419876,53839550,4590083968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:39:44,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 127, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":359174544,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3843761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":489438944,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":296086800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335110144,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.12],"systemCyclesUsage":[66419876,53839550,4590083968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:39:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:39:44,145 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-28:[ctx-a2b95791]) (logid:b04ab5c7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":359174544,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3843761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":489438944,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":296086800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335110144,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.12],"systemCyclesUsage":[66419876,53839550,4590083968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:39:44,145 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:39:44,145 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 127, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:39:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":359174544,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3843761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":489438944,"totalCommitted":852885504,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":296086800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335110144,"systemMemoryUsed":1019588,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.12],"systemCyclesUsage":[66419876,53839550,4590083968],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:39:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4eeb1f9]) (logid:007b66d5) No inactive management server node found 2025-06-29 04:39:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4eeb1f9]) (logid:007b66d5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:39:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:39:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-045d36b9]) (logid:90dd1f4c) No inactive management server node found 2025-06-29 04:39:46,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-045d36b9]) (logid:90dd1f4c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:39:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:39:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:39:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:39:46,670 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-33cf821f]) (logid:6ea3b562) HA health check task is running... 2025-06-29 04:39:47,103 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:39:47,105 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:39:47,106 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:39:47,108 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:39:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56616a2c]) (logid:b4250ec7) No inactive management server node found 2025-06-29 04:39:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56616a2c]) (logid:b4250ec7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-62d76ab7]) (logid:03904c27) Resetting hosts suitable for reconnect 2025-06-29 04:39:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-62d76ab7]) (logid:03904c27) Completed resetting hosts suitable for reconnect 2025-06-29 04:39:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-62d76ab7]) (logid:03904c27) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:39:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-62d76ab7]) (logid:03904c27) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:39:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-62d76ab7]) (logid:03904c27) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:39:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-62d76ab7]) (logid:03904c27) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:39:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-67d423fa]) (logid:8a7c59d9) VM scheduler.poll is being called at 2025-06-29T02:40:00+0000 2025-06-29 04:39:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-67d423fa]) (logid:8a7c59d9) Got 0 scheduled jobs to be executed at 2025-06-29T02:40:00+0000 2025-06-29 04:39:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-67d423fa]) (logid:8a7c59d9) Cleaned up 0 VM scheduled job entries 2025-06-29 04:39:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-28d424e2]) (logid:e89f1a1a) VM Operation Thread Running 2025-06-29 04:39:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-63d002e0]) (logid:df42f798) ClusterDRS.poll is being called at 2025-06-29T02:40:00+0000 2025-06-29 04:39:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-63d002e0]) (logid:df42f798) Removed 0 old drs migration plans 2025-06-29 04:39:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-300a8505]) (logid:2dc48977) No inactive management server node found 2025-06-29 04:39:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-300a8505]) (logid:2dc48977) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:50,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8710: Seq 7-8710: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:50,386 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8710 2025-06-29 04:39:50,671 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5b99d939]) (logid:1253f005) HA health check task is running... 2025-06-29 04:39:50,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61799657]) (logid:2b2fb738) No inactive management server node found 2025-06-29 04:39:50,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61799657]) (logid:2b2fb738) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83b88952]) (logid:555f8e40) No inactive management server node found 2025-06-29 04:39:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83b88952]) (logid:555f8e40) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb36c2bb]) (logid:1409d9f9) No inactive management server node found 2025-06-29 04:39:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb36c2bb]) (logid:1409d9f9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:54,672 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e6497103]) (logid:d4698675) HA health check task is running... 2025-06-29 04:39:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c5dad6c]) (logid:baa4b649) No inactive management server node found 2025-06-29 04:39:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c5dad6c]) (logid:baa4b649) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:55,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8711: Seq 7-8711: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:39:55,385 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8711 2025-06-29 04:39:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2606fb3]) (logid:9bd4bccf) No inactive management server node found 2025-06-29 04:39:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c2606fb3]) (logid:9bd4bccf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:39:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ed9a6602]) (logid:9c9b5d0b) Enabled non-edge zones available for scan: 1 2025-06-29 04:39:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-ed9a6602]) (logid:9c9b5d0b) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:39:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-ed9a6602]) (logid:9c9b5d0b) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:39:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-94f7c72a]) (logid:3c666c28) Enabled non-edge zones available for scan: 1 2025-06-29 04:39:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-94f7c72a]) (logid:3c666c28) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:39:58,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85ba51ac]) (logid:c38c0994) No inactive management server node found 2025-06-29 04:39:58,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85ba51ac]) (logid:c38c0994) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:39:58,673 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b880b143]) (logid:9492c39b) HA health check task is running... 2025-06-29 04:39:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fae6371]) (logid:9aff0b50) No inactive management server node found 2025-06-29 04:39:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fae6371]) (logid:9aff0b50) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bfd6086]) (logid:1c555548) No inactive management server node found 2025-06-29 04:40:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9bfd6086]) (logid:1c555548) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:02,675 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f604eee2]) (logid:527f9061) HA health check task is running... 2025-06-29 04:40:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e224446]) (logid:a8385636) No inactive management server node found 2025-06-29 04:40:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e224446]) (logid:a8385636) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6855bbe5]) (logid:ed720d8d) No inactive management server node found 2025-06-29 04:40:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6855bbe5]) (logid:ed720d8d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:05,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8712: Seq 7-8712: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:40:05,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8712 2025-06-29 04:40:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d887b05]) (logid:7222391d) No inactive management server node found 2025-06-29 04:40:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d887b05]) (logid:7222391d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:06,676 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a386de2d]) (logid:a21503c7) HA health check task is running... 2025-06-29 04:40:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1728ace]) (logid:ba78dd96) No inactive management server node found 2025-06-29 04:40:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e1728ace]) (logid:ba78dd96) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ac0f4ff]) (logid:f3819512) No inactive management server node found 2025-06-29 04:40:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ac0f4ff]) (logid:f3819512) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f73e3196]) (logid:482898be) No inactive management server node found 2025-06-29 04:40:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f73e3196]) (logid:482898be) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:10,677 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-fc1b7af3]) (logid:0c3f1550) HA health check task is running... 2025-06-29 04:40:11,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a0fcbad]) (logid:b8ca58d7) No inactive management server node found 2025-06-29 04:40:11,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a0fcbad]) (logid:b8ca58d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-37a796dd]) (logid:7be17be1) Found 0 routers to update status. 2025-06-29 04:40:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-37a796dd]) (logid:7be17be1) Found 0 VPC's to update Redundant State. 2025-06-29 04:40:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-37a796dd]) (logid:7be17be1) Found 0 networks to update RvR status. 2025-06-29 04:40:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8759180]) (logid:e9e57f8e) No inactive management server node found 2025-06-29 04:40:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8759180]) (logid:e9e57f8e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:14,678 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-4267d228]) (logid:c1476a52) HA health check task is running... 2025-06-29 04:40:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11e83e61]) (logid:05152be4) No inactive management server node found 2025-06-29 04:40:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11e83e61]) (logid:05152be4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:15,375 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8713: Seq 7-8713: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:40:15,386 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8713 2025-06-29 04:40:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21a5344d]) (logid:5398b8b0) No inactive management server node found 2025-06-29 04:40:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21a5344d]) (logid:5398b8b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:17,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cef179b]) (logid:9b66952a) No inactive management server node found 2025-06-29 04:40:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cef179b]) (logid:9b66952a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:18,065 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-73956427]) (logid:b389a3f7) AutoScaling Monitor is running... 2025-06-29 04:40:18,138 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-66d36db4]) (logid:89eb0b77) Starting to remove all volume_stats rows older than [Sat Jun 28 16:40:18 CEST 2025]. 2025-06-29 04:40:18,138 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-66d36db4]) (logid:89eb0b77) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:40:18 CEST 2025]. 2025-06-29 04:40:18,680 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0367efff]) (logid:144f40e3) HA health check task is running... 2025-06-29 04:40:18,980 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-d6bc276c]) (logid:46df656a) Starting to remove all vm_stats rows older than [Sat Jun 28 16:40:18 CEST 2025]. 2025-06-29 04:40:19,006 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-d6bc276c]) (logid:46df656a) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:40:18 CEST 2025]. 2025-06-29 04:40:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-669094ab]) (logid:b79f2295) No inactive management server node found 2025-06-29 04:40:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-669094ab]) (logid:b79f2295) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97247dab]) (logid:509a3364) No inactive management server node found 2025-06-29 04:40:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97247dab]) (logid:509a3364) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:21,293 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:40:21,295 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:40:21,296 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) Seq 2-4725683383995203849: Routed from 90520738888109 2025-06-29 04:40:21,339 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) Seq 1-4725683383995203849: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:40:21,341 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:40:21,342 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) Seq 2-3926857400090362152: Routed from 90520738888109 2025-06-29 04:40:21,386 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-86550e29]) (logid:aafc3c56) Seq 2-3926857400090362152: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:40:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dcf8fde]) (logid:302dbcf4) No inactive management server node found 2025-06-29 04:40:22,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0dcf8fde]) (logid:302dbcf4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:22,681 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-47fd3df5]) (logid:054713df) HA health check task is running... 2025-06-29 04:40:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f38f2f8]) (logid:a1ea5c88) No inactive management server node found 2025-06-29 04:40:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f38f2f8]) (logid:a1ea5c88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:25,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aaebbe6c]) (logid:1769ad21) No inactive management server node found 2025-06-29 04:40:25,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aaebbe6c]) (logid:1769ad21) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:25,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8714: Seq 7-8714: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:40:25,387 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8714 2025-06-29 04:40:26,682 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9afe04dd]) (logid:ef29de50) HA health check task is running... 2025-06-29 04:40:26,763 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:40:26,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:40:26,768 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) Seq 2-4725683383995203850: Routed from 90520738888109 2025-06-29 04:40:26,821 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) Seq 1-4725683383995203850: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:40:26,854 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:40:26,856 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) Seq 2-3926857400090362153: Routed from 90520738888109 2025-06-29 04:40:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57da878f]) (logid:95bbea3e) No inactive management server node found 2025-06-29 04:40:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57da878f]) (logid:95bbea3e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:26,954 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-19f26857]) (logid:25083548) Seq 2-3926857400090362153: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:40:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-cfba7317]) (logid:17edfc09) Enabled non-edge zones available for scan: 1 2025-06-29 04:40:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-cfba7317]) (logid:17edfc09) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:40:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-cfba7317]) (logid:17edfc09) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:40:28,209 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) StorageCollector is running... 2025-06-29 04:40:28,214 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:40:28,214 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:40:28,214 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:40:28,215 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-3938116399158788167: Routed from 90520738888109 2025-06-29 04:40:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d8622310]) (logid:3977fea0) Enabled non-edge zones available for scan: 1 2025-06-29 04:40:28,283 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d8622310]) (logid:3977fea0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:40:28,292 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 4-3938116399158788167: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:40:28,297 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:40:28,297 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:40:28,297 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:40:28,298 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-4725683383995203851: Routed from 90520738888109 2025-06-29 04:40:28,345 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 1-4725683383995203851: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:40:28,357 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:40:28,358 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:40:28,358 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:40:28,359 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-3926857400090362154: Routed from 90520738888109 2025-06-29 04:40:28,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18082722]) (logid:8fc743b2) No inactive management server node found 2025-06-29 04:40:28,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18082722]) (logid:8fc743b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:28,370 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-3926857400090362154: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:40:28,380 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:40:28,380 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:40:28,380 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:40:28,381 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-4725683383995203852: Routed from 90520738888109 2025-06-29 04:40:28,466 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 1-4725683383995203852: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:40:28,469 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:40:28,469 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:40:28,469 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:40:28,470 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-3926857400090362155: Routed from 90520738888109 2025-06-29 04:40:28,514 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-e0cd69c9]) (logid:f1e53731) Seq 2-3926857400090362155: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:40:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6c15de2]) (logid:4d709bfc) No inactive management server node found 2025-06-29 04:40:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c6c15de2]) (logid:4d709bfc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:30,684 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0639e31b]) (logid:fca59fb1) HA health check task is running... 2025-06-29 04:40:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88f33d58]) (logid:0d22db34) No inactive management server node found 2025-06-29 04:40:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-88f33d58]) (logid:0d22db34) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ad5edef]) (logid:5078a360) No inactive management server node found 2025-06-29 04:40:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ad5edef]) (logid:5078a360) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99f3e55e]) (logid:86b9a270) No inactive management server node found 2025-06-29 04:40:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99f3e55e]) (logid:86b9a270) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:34,685 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e3d8aef5]) (logid:8f15e24a) HA health check task is running... 2025-06-29 04:40:35,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8715: Seq 7-8715: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:40:35,387 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8715 2025-06-29 04:40:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06d2fb52]) (logid:3779617e) No inactive management server node found 2025-06-29 04:40:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06d2fb52]) (logid:3779617e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-710b41e6]) (logid:cf1c962a) No inactive management server node found 2025-06-29 04:40:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-710b41e6]) (logid:cf1c962a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:38,687 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8c7ed05d]) (logid:e902ea82) HA health check task is running... 2025-06-29 04:40:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-991257d0]) (logid:2ee9b1a1) No inactive management server node found 2025-06-29 04:40:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-991257d0]) (logid:2ee9b1a1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-860346a5]) (logid:280b1504) No inactive management server node found 2025-06-29 04:40:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-860346a5]) (logid:280b1504) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edae18a2]) (logid:38c8ddde) No inactive management server node found 2025-06-29 04:40:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edae18a2]) (logid:38c8ddde) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:42,688 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d8454ffe]) (logid:879766ca) HA health check task is running... 2025-06-29 04:40:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-c6449a4f]) (logid:01a37d03) DbCollector is running... 2025-06-29 04:40:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2929d7a1]) (logid:dff933bc) Found 0 routers to update status. 2025-06-29 04:40:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2929d7a1]) (logid:dff933bc) Found 0 VPC's to update Redundant State. 2025-06-29 04:40:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2929d7a1]) (logid:dff933bc) Found 0 networks to update RvR status. 2025-06-29 04:40:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7ed5e72]) (logid:b60b3835) No inactive management server node found 2025-06-29 04:40:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7ed5e72]) (logid:b60b3835) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) ManagementServerCollector is running... 2025-06-29 04:40:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) DB properties were already loaded 2025-06-29 04:40:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:40:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Execution of process [4067586] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:40:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Exit value of process [4067586] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:40:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Process [4067586] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:40:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) usage install: null 2025-06-29 04:40:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Sessions active: 0 2025-06-29 04:40:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:40:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067589] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:40:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:40:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067594] for command [/bin/bash -c uname -r ]. 2025-06-29 04:40:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:40:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067595] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:40:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) system memory from /proc: 4887515136 2025-06-29 04:40:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:40:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067600] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:40:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) free memory from /proc: 332746752 2025-06-29 04:40:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:40:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067605] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:40:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) used memory from /proc: 1019716 2025-06-29 04:40:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:40:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067606] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:40:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:40:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067607] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:40:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:40:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067608] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:40:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:40:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067615] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:40:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:40:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Successfully executed process [4067616] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail buffersdirect.capacity, 326696 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail buffersdirect.count, 96 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail buffersdirect.used, 326696 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail buffersmapped.capacity, 0 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail buffersmapped.count, 0 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail buffersmapped.used, 0 2025-06-29 04:40:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail jvmuptime, 3903831 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memoryheap.committed, 654311424 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memoryheap.init, 77594624 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memoryheap.usage, 0.16734028150667957 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorynon-heap.committed, 198639616 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorynon-heap.max, -1 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorynon-heap.usage, -1.945342E8 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorynon-heap.used, 194534200 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11555668488403092 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21635576698770287 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388065457344055 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.Metaspace.usage, 0.9928820577377112 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.PS-Eden-Space.usage, 0.1942039077844858 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorytotal.committed, 852951040 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorytotal.init, 85262336 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorytotal.max, 1908932607 2025-06-29 04:40:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail memorytotal.used, 513978464 2025-06-29 04:40:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail threadsdeadlocks, [] 2025-06-29 04:40:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail threadsnew.count, 0 2025-06-29 04:40:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:40:44,132 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-70e67a49]) (logid:90b38a4b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":335559176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3903760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513285976,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":319441320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":332746752,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.11],"systemCyclesUsage":[66420201,53839846,4590107100],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:40:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 129, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":335559176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3903760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513285976,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":319441320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":332746752,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.11],"systemCyclesUsage":[66420201,53839846,4590107100],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:40:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:40:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-28:[ctx-a483103c]) (logid:3eaaa324) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":335559176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3903760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513285976,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":319441320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":332746752,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.11],"systemCyclesUsage":[66420201,53839846,4590107100],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:40:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:40:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 7ms. agent: 0, pdu seq: 129, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:40:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":335559176,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3903760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":513285976,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":319441320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":332746752,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.02,0.11],"systemCyclesUsage":[66420201,53839846,4590107100],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:40:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb4343b5]) (logid:ceeb751f) No inactive management server node found 2025-06-29 04:40:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb4343b5]) (logid:ceeb751f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:45,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8716: Seq 7-8716: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:40:45,387 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8716 2025-06-29 04:40:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:40:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:40:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4835bc5]) (logid:208d8d60) No inactive management server node found 2025-06-29 04:40:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4835bc5]) (logid:208d8d60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:40:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:40:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:40:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:40:46,689 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5a7481ef]) (logid:24380b1d) HA health check task is running... 2025-06-29 04:40:47,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:40:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:40:47,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:40:47,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:40:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55816e5a]) (logid:42475239) No inactive management server node found 2025-06-29 04:40:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55816e5a]) (logid:42475239) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2dd00dde]) (logid:56cfa08e) VM scheduler.poll is being called at 2025-06-29T02:41:00+0000 2025-06-29 04:40:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2dd00dde]) (logid:56cfa08e) Got 0 scheduled jobs to be executed at 2025-06-29T02:41:00+0000 2025-06-29 04:40:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-2dd00dde]) (logid:56cfa08e) Cleaned up 0 VM scheduled job entries 2025-06-29 04:40:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-3aef5b8f]) (logid:f4c8ca0a) VM Operation Thread Running 2025-06-29 04:40:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6438b2d6]) (logid:a008e3c0) ClusterDRS.poll is being called at 2025-06-29T02:41:00+0000 2025-06-29 04:40:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6438b2d6]) (logid:a008e3c0) Removed 0 old drs migration plans 2025-06-29 04:40:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-258dd4ab]) (logid:e26c8b22) No inactive management server node found 2025-06-29 04:40:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-258dd4ab]) (logid:e26c8b22) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:50,690 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1832cc03]) (logid:fefd4157) HA health check task is running... 2025-06-29 04:40:50,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43dcc6b2]) (logid:732e9f04) No inactive management server node found 2025-06-29 04:40:50,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-43dcc6b2]) (logid:732e9f04) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2317d2e3]) (logid:ebf1d3a5) No inactive management server node found 2025-06-29 04:40:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2317d2e3]) (logid:ebf1d3a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7e6c1d5]) (logid:a8871e12) No inactive management server node found 2025-06-29 04:40:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7e6c1d5]) (logid:a8871e12) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:54,691 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c2268dd8]) (logid:f5b2beec) HA health check task is running... 2025-06-29 04:40:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62302f8a]) (logid:75d4ec9c) No inactive management server node found 2025-06-29 04:40:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-62302f8a]) (logid:75d4ec9c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:55,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8718: Seq 7-8718: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:40:55,387 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8718 2025-06-29 04:40:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b834989]) (logid:c4d4d404) No inactive management server node found 2025-06-29 04:40:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b834989]) (logid:c4d4d404) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d5e97464]) (logid:82146d87) Enabled non-edge zones available for scan: 1 2025-06-29 04:40:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d5e97464]) (logid:82146d87) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:40:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d5e97464]) (logid:82146d87) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:40:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3755216c]) (logid:ad8a1bcd) Enabled non-edge zones available for scan: 1 2025-06-29 04:40:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3755216c]) (logid:ad8a1bcd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:40:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6f1fdc4]) (logid:fb67a528) No inactive management server node found 2025-06-29 04:40:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6f1fdc4]) (logid:fb67a528) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:40:58,693 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-f270732e]) (logid:985c7bd8) HA health check task is running... 2025-06-29 04:40:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcb75eb5]) (logid:a1a2ee60) No inactive management server node found 2025-06-29 04:40:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcb75eb5]) (logid:a1a2ee60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:41:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-811db856]) (logid:c4f9f659) No inactive management server node found 2025-06-29 04:41:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-811db856]) (logid:c4f9f659) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:02,694 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0c88b4ab]) (logid:53a94333) HA health check task is running... 2025-06-29 04:41:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e5dab20]) (logid:a335d46a) No inactive management server node found 2025-06-29 04:41:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e5dab20]) (logid:a335d46a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31abe806]) (logid:54493db7) No inactive management server node found 2025-06-29 04:41:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-31abe806]) (logid:54493db7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:05,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8719: Seq 7-8719: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:41:05,388 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8719 2025-06-29 04:41:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7def2f7]) (logid:0d749ebb) No inactive management server node found 2025-06-29 04:41:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7def2f7]) (logid:0d749ebb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:06,695 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-edcb874e]) (logid:ffd22f6b) HA health check task is running... 2025-06-29 04:41:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af2cda01]) (logid:fbefc014) No inactive management server node found 2025-06-29 04:41:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af2cda01]) (logid:fbefc014) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbce879b]) (logid:a254e3e7) No inactive management server node found 2025-06-29 04:41:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbce879b]) (logid:a254e3e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a16ea755]) (logid:bec7a3a9) No inactive management server node found 2025-06-29 04:41:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a16ea755]) (logid:bec7a3a9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:10,696 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-79c8bf09]) (logid:ccd7a58f) HA health check task is running... 2025-06-29 04:41:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a78101a]) (logid:cb4f52c9) No inactive management server node found 2025-06-29 04:41:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a78101a]) (logid:cb4f52c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c5ff5cfb]) (logid:8d18608d) Found 0 routers to update status. 2025-06-29 04:41:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c5ff5cfb]) (logid:8d18608d) Found 0 VPC's to update Redundant State. 2025-06-29 04:41:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c5ff5cfb]) (logid:8d18608d) Found 0 networks to update RvR status. 2025-06-29 04:41:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56c101ce]) (logid:6635a2da) No inactive management server node found 2025-06-29 04:41:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56c101ce]) (logid:6635a2da) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:14,698 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-256e9cbd]) (logid:60d00a40) HA health check task is running... 2025-06-29 04:41:14,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d301bbc5]) (logid:306a7f97) No inactive management server node found 2025-06-29 04:41:14,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d301bbc5]) (logid:306a7f97) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:15,378 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8720: Seq 7-8720: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:41:15,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8720 2025-06-29 04:41:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-119dc753]) (logid:3c4875c8) No inactive management server node found 2025-06-29 04:41:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-119dc753]) (logid:3c4875c8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb089355]) (logid:c6e04046) No inactive management server node found 2025-06-29 04:41:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb089355]) (logid:c6e04046) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:18,067 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-c2d69568]) (logid:a818af86) AutoScaling Monitor is running... 2025-06-29 04:41:18,093 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-3118c17c]) (logid:255cf020) Resetting hosts suitable for reconnect 2025-06-29 04:41:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-3118c17c]) (logid:255cf020) Completed resetting hosts suitable for reconnect 2025-06-29 04:41:18,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-3118c17c]) (logid:255cf020) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:41:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-3118c17c]) (logid:255cf020) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:41:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-3118c17c]) (logid:255cf020) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:41:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-3118c17c]) (logid:255cf020) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:41:18,140 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-f7ef53c9]) (logid:cbc0adfe) Starting to remove all volume_stats rows older than [Sat Jun 28 16:41:18 CEST 2025]. 2025-06-29 04:41:18,141 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-f7ef53c9]) (logid:cbc0adfe) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:41:18 CEST 2025]. 2025-06-29 04:41:18,699 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-67476ebe]) (logid:238582bf) HA health check task is running... 2025-06-29 04:41:19,007 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-dae5fb90]) (logid:20e150ce) Starting to remove all vm_stats rows older than [Sat Jun 28 16:41:19 CEST 2025]. 2025-06-29 04:41:19,031 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-dae5fb90]) (logid:20e150ce) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:41:19 CEST 2025]. 2025-06-29 04:41:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c34559a]) (logid:fe9e457d) No inactive management server node found 2025-06-29 04:41:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c34559a]) (logid:fe9e457d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:20,379 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8721: Seq 7-8721: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:41:20,388 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8721 2025-06-29 04:41:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb562117]) (logid:d0eaf5d6) No inactive management server node found 2025-06-29 04:41:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb562117]) (logid:d0eaf5d6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:21,388 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:41:21,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:41:21,391 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) Seq 2-4725683383995203853: Routed from 90520738888109 2025-06-29 04:41:21,434 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) Seq 1-4725683383995203853: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:41:21,436 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:41:21,437 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) Seq 2-3926857400090362156: Routed from 90520738888109 2025-06-29 04:41:21,480 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-46adf692]) (logid:318d2757) Seq 2-3926857400090362156: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:41:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e462db63]) (logid:50d74c3c) No inactive management server node found 2025-06-29 04:41:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e462db63]) (logid:50d74c3c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:22,700 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-72e14f93]) (logid:2b5ca1f8) HA health check task is running... 2025-06-29 04:41:23,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aedad32b]) (logid:e10ed3f1) No inactive management server node found 2025-06-29 04:41:23,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aedad32b]) (logid:e10ed3f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-270ce74a]) (logid:38696d9f) No inactive management server node found 2025-06-29 04:41:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-270ce74a]) (logid:38696d9f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:26,701 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0049e332]) (logid:b4718e5a) HA health check task is running... 2025-06-29 04:41:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe8d817b]) (logid:1fd3b404) No inactive management server node found 2025-06-29 04:41:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe8d817b]) (logid:1fd3b404) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:27,010 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:41:27,013 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:41:27,014 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) Seq 2-4725683383995203854: Routed from 90520738888109 2025-06-29 04:41:27,071 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) Seq 1-4725683383995203854: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:41:27,106 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:41:27,107 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) Seq 2-3926857400090362157: Routed from 90520738888109 2025-06-29 04:41:27,212 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-d0e02ede]) (logid:6969ee5d) Seq 2-3926857400090362157: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:41:28,090 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8b87a572]) (logid:2d4163ff) Enabled non-edge zones available for scan: 1 2025-06-29 04:41:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8b87a572]) (logid:2d4163ff) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:41:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8b87a572]) (logid:2d4163ff) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:41:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f9ae481c]) (logid:e86ee55a) Enabled non-edge zones available for scan: 1 2025-06-29 04:41:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f9ae481c]) (logid:e86ee55a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:41:28,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10931688]) (logid:e818746c) No inactive management server node found 2025-06-29 04:41:28,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10931688]) (logid:e818746c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:28,515 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) StorageCollector is running... 2025-06-29 04:41:28,521 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:41:28,521 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:41:28,521 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:41:28,523 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-3938116399158788168: Routed from 90520738888109 2025-06-29 04:41:28,576 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 4-3938116399158788168: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:41:28,580 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:41:28,580 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:41:28,580 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:41:28,581 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-3926857400090362158: Routed from 90520738888109 2025-06-29 04:41:28,593 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-3926857400090362158: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:41:28,604 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:41:28,604 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:41:28,604 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:41:28,605 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-4725683383995203855: Routed from 90520738888109 2025-06-29 04:41:28,654 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 1-4725683383995203855: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:41:28,671 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:41:28,671 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:41:28,671 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:41:28,672 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-4725683383995203856: Routed from 90520738888109 2025-06-29 04:41:28,756 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 1-4725683383995203856: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:41:28,758 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:41:28,758 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:41:28,758 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:41:28,759 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-3926857400090362159: Routed from 90520738888109 2025-06-29 04:41:28,804 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-a22a07f5]) (logid:5787d959) Seq 2-3926857400090362159: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:41:29,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79ad8b25]) (logid:1c647712) No inactive management server node found 2025-06-29 04:41:29,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-79ad8b25]) (logid:1c647712) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:41:30,379 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8722: Seq 7-8722: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:41:30,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8722 2025-06-29 04:41:30,703 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-18aafd42]) (logid:1e2146bc) HA health check task is running... 2025-06-29 04:41:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-450af6f8]) (logid:9a2e03c4) No inactive management server node found 2025-06-29 04:41:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-450af6f8]) (logid:9a2e03c4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-753abb2f]) (logid:8373c455) No inactive management server node found 2025-06-29 04:41:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-753abb2f]) (logid:8373c455) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67ec029f]) (logid:188e68a8) No inactive management server node found 2025-06-29 04:41:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67ec029f]) (logid:188e68a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:34,704 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-1a8e3510]) (logid:92b4a0e9) HA health check task is running... 2025-06-29 04:41:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f4f0f7]) (logid:ac28dd79) No inactive management server node found 2025-06-29 04:41:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f4f0f7]) (logid:ac28dd79) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-834b610a]) (logid:03321834) No inactive management server node found 2025-06-29 04:41:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-834b610a]) (logid:03321834) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:38,705 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b5b2e780]) (logid:9084bcae) HA health check task is running... 2025-06-29 04:41:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-711fd603]) (logid:36bf0494) No inactive management server node found 2025-06-29 04:41:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-711fd603]) (logid:36bf0494) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8f23a33]) (logid:f860b155) No inactive management server node found 2025-06-29 04:41:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8f23a33]) (logid:f860b155) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:40,379 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8723: Seq 7-8723: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:41:40,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8723 2025-06-29 04:41:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecb095ca]) (logid:f61a2f5d) No inactive management server node found 2025-06-29 04:41:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecb095ca]) (logid:f61a2f5d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:42,706 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5ffb24ef]) (logid:cf79ae96) HA health check task is running... 2025-06-29 04:41:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-d686229a]) (logid:25b020fd) DbCollector is running... 2025-06-29 04:41:43,095 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-81e877a5]) (logid:b72ccbac) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:41:43,096 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-81e877a5]) (logid:b72ccbac) Snapshot scheduler is being called at [2025-06-29T02:41:43+0000]. 2025-06-29 04:41:43,097 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-81e877a5]) (logid:b72ccbac) There are [0] scheduled snapshots to be executed at [2025-06-29T02:41:43+0000]. 2025-06-29 04:41:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-dec5c44d]) (logid:49f38f2e) Found 0 running routers. 2025-06-29 04:41:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8cfb5ceb]) (logid:02f10e1c) Found 0 routers to update status. 2025-06-29 04:41:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8cfb5ceb]) (logid:02f10e1c) Found 0 VPC's to update Redundant State. 2025-06-29 04:41:43,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8cfb5ceb]) (logid:02f10e1c) Found 0 networks to update RvR status. 2025-06-29 04:41:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-0a6e5316]) (logid:6c2c42ed) External devices are not used. Skipping external device usage collection 2025-06-29 04:41:43,137 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-9fc69b94]) (logid:5f1c77b4) Started resource counters recalculation periodic task. 2025-06-29 04:41:43,140 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-9fc69b94]) (logid:5f1c77b4) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:41:43,306 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-f28c1b70]) (logid:41d93175) Backup backup.poll is being called at 2025-06-29T02:41:43+0000 2025-06-29 04:41:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3413c379]) (logid:6885161a) No inactive management server node found 2025-06-29 04:41:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3413c379]) (logid:6885161a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:43,527 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-9fc69b94]) (logid:5f1c77b4) Finished resource counters recalculation periodic task. 2025-06-29 04:41:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) ManagementServerCollector is running... 2025-06-29 04:41:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) DB properties were already loaded 2025-06-29 04:41:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:41:44,059 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Execution of process [4067621] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:41:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Exit value of process [4067621] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:41:44,059 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Process [4067621] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:41:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) usage install: null 2025-06-29 04:41:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Sessions active: 0 2025-06-29 04:41:44,062 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:41:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067624] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:41:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:41:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067629] for command [/bin/bash -c uname -r ]. 2025-06-29 04:41:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:41:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067630] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:41:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) system memory from /proc: 4887515136 2025-06-29 04:41:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:41:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067635] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:41:44,084 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) free memory from /proc: 335556608 2025-06-29 04:41:44,084 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:41:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067640] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:41:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) used memory from /proc: 1019716 2025-06-29 04:41:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:41:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067641] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:41:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:41:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067642] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:41:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:41:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067643] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:41:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:41:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067650] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:41:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:41:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Successfully executed process [4067651] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail buffersdirect.capacity, 326696 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail buffersdirect.count, 96 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail buffersdirect.used, 326696 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail buffersmapped.capacity, 0 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail buffersmapped.count, 0 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail buffersmapped.used, 0 2025-06-29 04:41:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail jvmuptime, 3963833 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memoryheap.committed, 654311424 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memoryheap.init, 77594624 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memoryheap.usage, 0.18473798316509243 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorynon-heap.committed, 198639616 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorynon-heap.max, -1 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorynon-heap.usage, -1.9475828E8 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorynon-heap.used, 194758280 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11591804685417222 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2175013330222948 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388065457344055 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.Metaspace.usage, 0.9931392589123358 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.PS-Eden-Space.usage, 0.24524744665497836 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorytotal.committed, 852951040 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorytotal.init, 85262336 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorytotal.max, 1908932607 2025-06-29 04:41:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail memorytotal.used, 547410640 2025-06-29 04:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail threadsdeadlocks, [] 2025-06-29 04:41:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail threadsnew.count, 0 2025-06-29 04:41:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:41:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-1046fc5b]) (logid:ac6858a2) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":302527024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3963763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":546542680,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":352652360,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335556608,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.1],"systemCyclesUsage":[66420566,53840149,4590130177],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:41:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 131, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":302527024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3963763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":546542680,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":352652360,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335556608,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.1],"systemCyclesUsage":[66420566,53840149,4590130177],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:41:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:41:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-28:[ctx-3d9af2bc]) (logid:3622a23b) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":302527024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3963763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":546542680,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":352652360,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335556608,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.1],"systemCyclesUsage":[66420566,53840149,4590130177],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:41:44,139 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:41:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 131, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:41:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":302527024,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":3963763,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.0,"totalInit":85262336,"totalUsed":546542680,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":352652360,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":335556608,"systemMemoryUsed":1019716,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.0,0.01,0.1],"systemCyclesUsage":[66420566,53840149,4590130177],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:41:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-202c8257]) (logid:6a762043) No inactive management server node found 2025-06-29 04:41:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-202c8257]) (logid:6a762043) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:41:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:41:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d497dcce]) (logid:1335afbb) No inactive management server node found 2025-06-29 04:41:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d497dcce]) (logid:1335afbb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:41:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:41:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:41:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:41:46,707 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-37a9d288]) (logid:2da824ff) HA health check task is running... 2025-06-29 04:41:47,086 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:41:47,088 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:41:47,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:41:47,091 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:41:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d346234d]) (logid:12535c21) No inactive management server node found 2025-06-29 04:41:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d346234d]) (logid:12535c21) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:48,057 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-3:[ctx-a9eed141]) (logid:159d0920) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:41:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-193b0ba3]) (logid:0a6c97c0) VM scheduler.poll is being called at 2025-06-29T02:42:00+0000 2025-06-29 04:41:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-193b0ba3]) (logid:0a6c97c0) Got 0 scheduled jobs to be executed at 2025-06-29T02:42:00+0000 2025-06-29 04:41:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-193b0ba3]) (logid:0a6c97c0) Cleaned up 0 VM scheduled job entries 2025-06-29 04:41:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9970004d]) (logid:10ea8339) VM Operation Thread Running 2025-06-29 04:41:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6449b5b7]) (logid:073a1c24) ClusterDRS.poll is being called at 2025-06-29T02:42:00+0000 2025-06-29 04:41:48,357 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6449b5b7]) (logid:073a1c24) Removed 0 old drs migration plans 2025-06-29 04:41:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3605d88d]) (logid:d4ab19e9) No inactive management server node found 2025-06-29 04:41:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3605d88d]) (logid:d4ab19e9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:50,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8725: Seq 7-8725: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:41:50,391 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8725 2025-06-29 04:41:50,709 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f8fefb1a]) (logid:506a79f2) HA health check task is running... 2025-06-29 04:41:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46683fa0]) (logid:5a0c2eb2) No inactive management server node found 2025-06-29 04:41:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46683fa0]) (logid:5a0c2eb2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fc69d27]) (logid:1d01f3e2) No inactive management server node found 2025-06-29 04:41:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3fc69d27]) (logid:1d01f3e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9acd40e0]) (logid:a67efc4a) No inactive management server node found 2025-06-29 04:41:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9acd40e0]) (logid:a67efc4a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:54,710 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2415dc45]) (logid:1d40e30b) HA health check task is running... 2025-06-29 04:41:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5fbeab0]) (logid:b76359bb) No inactive management server node found 2025-06-29 04:41:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5fbeab0]) (logid:b76359bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7a1669b]) (logid:b037375c) No inactive management server node found 2025-06-29 04:41:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7a1669b]) (logid:b037375c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2a3ad2dc]) (logid:b1e5a5c1) Enabled non-edge zones available for scan: 1 2025-06-29 04:41:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-2a3ad2dc]) (logid:b1e5a5c1) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:41:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-2a3ad2dc]) (logid:b1e5a5c1) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:41:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-07112d61]) (logid:3b05efd8) Enabled non-edge zones available for scan: 1 2025-06-29 04:41:58,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-07112d61]) (logid:3b05efd8) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:41:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba3c9fb7]) (logid:f42b4616) No inactive management server node found 2025-06-29 04:41:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba3c9fb7]) (logid:f42b4616) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:41:58,711 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-2bda2283]) (logid:82b45c33) HA health check task is running... 2025-06-29 04:41:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f566adb]) (logid:ac377196) No inactive management server node found 2025-06-29 04:41:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f566adb]) (logid:ac377196) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:00,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8726: Seq 7-8726: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:00,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8726 2025-06-29 04:42:01,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d79b6099]) (logid:b86e376b) No inactive management server node found 2025-06-29 04:42:01,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d79b6099]) (logid:b86e376b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:02,712 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ae695563]) (logid:de608d11) HA health check task is running... 2025-06-29 04:42:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b302719]) (logid:64dac659) No inactive management server node found 2025-06-29 04:42:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b302719]) (logid:64dac659) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b99b63ed]) (logid:5b4f412e) No inactive management server node found 2025-06-29 04:42:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b99b63ed]) (logid:5b4f412e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:05,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8727: Seq 7-8727: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:05,392 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8727 2025-06-29 04:42:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19ef5000]) (logid:2242d78d) No inactive management server node found 2025-06-29 04:42:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-19ef5000]) (logid:2242d78d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:06,714 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-015991a7]) (logid:1b72b6b8) HA health check task is running... 2025-06-29 04:42:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ead5a72]) (logid:6165958f) No inactive management server node found 2025-06-29 04:42:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ead5a72]) (logid:6165958f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6822d938]) (logid:606259f1) No inactive management server node found 2025-06-29 04:42:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6822d938]) (logid:606259f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6af4c160]) (logid:d9be35d5) No inactive management server node found 2025-06-29 04:42:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6af4c160]) (logid:d9be35d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:10,715 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-898fa3be]) (logid:d3097c22) HA health check task is running... 2025-06-29 04:42:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95182628]) (logid:d21080c7) No inactive management server node found 2025-06-29 04:42:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95182628]) (logid:d21080c7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-42bd57ed]) (logid:465469bc) Checking if any host reservation can be released ... 2025-06-29 04:42:13,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-42bd57ed]) (logid:465469bc) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:13,017 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-42bd57ed]) (logid:465469bc) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:13,018 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-42bd57ed]) (logid:465469bc) Done running HostReservationReleaseChecker ... 2025-06-29 04:42:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2bf1de46]) (logid:7e7e5e3d) Found 0 routers to update status. 2025-06-29 04:42:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2bf1de46]) (logid:7e7e5e3d) Found 0 VPC's to update Redundant State. 2025-06-29 04:42:13,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2bf1de46]) (logid:7e7e5e3d) Found 0 networks to update RvR status. 2025-06-29 04:42:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Running Capacity Checker ... 2025-06-29 04:42:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) recalculating system capacity 2025-06-29 04:42:13,166 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Executing cpu/ram capacity update 2025-06-29 04:42:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:13,174 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:13,194 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:42:13,194 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:42:13,195 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:42:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:13,210 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:13,217 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:13,219 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:42:13,220 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:42:13,220 DEBUG [c.c.c.CapacityManagerImpl] (pool-38-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:42:13,229 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Done executing cpu/ram capacity update 2025-06-29 04:42:13,229 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Executing storage capacity update 2025-06-29 04:42:13,242 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:42:13,242 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:42:13,242 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:42:13,245 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:42:13,252 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:42:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:42:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:42:13,255 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:42:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:42:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:42:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:42:13,269 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:42:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:42:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:42:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:42:13,283 DEBUG [c.c.s.StorageManagerImpl] (pool-39-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:42:13,283 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Done executing storage capacity update 2025-06-29 04:42:13,283 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Executing capacity updates for public ip and Vlans 2025-06-29 04:42:13,299 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Done capacity updates for public ip and Vlans 2025-06-29 04:42:13,299 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Executing capacity updates for private ip 2025-06-29 04:42:13,303 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Done executing capacity updates for private ip 2025-06-29 04:42:13,303 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Done recalculating system capacity 2025-06-29 04:42:13,345 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-c44ce7df]) (logid:a5c4a5e2) Done running Capacity Checker ... 2025-06-29 04:42:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f7c3c25]) (logid:5ccab0ce) No inactive management server node found 2025-06-29 04:42:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f7c3c25]) (logid:5ccab0ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:14,716 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f3b1577f]) (logid:c9311ec5) HA health check task is running... 2025-06-29 04:42:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4bc8125]) (logid:98762b38) No inactive management server node found 2025-06-29 04:42:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4bc8125]) (logid:98762b38) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:15,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8728: Seq 7-8728: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:15,391 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8728 2025-06-29 04:42:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fd4d927]) (logid:b49ecb45) No inactive management server node found 2025-06-29 04:42:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fd4d927]) (logid:b49ecb45) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5438c800]) (logid:fc142e1a) No inactive management server node found 2025-06-29 04:42:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5438c800]) (logid:fc142e1a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:18,069 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-a2ea7ca3]) (logid:7721e739) AutoScaling Monitor is running... 2025-06-29 04:42:18,143 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-de91e2c5]) (logid:30c37d06) Starting to remove all volume_stats rows older than [Sat Jun 28 16:42:18 CEST 2025]. 2025-06-29 04:42:18,143 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-de91e2c5]) (logid:30c37d06) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:42:18 CEST 2025]. 2025-06-29 04:42:18,717 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9a48c7b5]) (logid:2387db87) HA health check task is running... 2025-06-29 04:42:19,032 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-5372d7cf]) (logid:a6113771) Starting to remove all vm_stats rows older than [Sat Jun 28 16:42:19 CEST 2025]. 2025-06-29 04:42:19,056 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-5372d7cf]) (logid:a6113771) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:42:19 CEST 2025]. 2025-06-29 04:42:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98d4cc03]) (logid:87e21094) No inactive management server node found 2025-06-29 04:42:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98d4cc03]) (logid:87e21094) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffb048f6]) (logid:0832840c) No inactive management server node found 2025-06-29 04:42:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffb048f6]) (logid:0832840c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:21,484 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:42:21,485 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:42:21,487 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) Seq 2-4725683383995203857: Routed from 90520738888109 2025-06-29 04:42:21,530 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) Seq 1-4725683383995203857: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:42:21,531 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:42:21,531 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) Seq 2-3926857400090362160: Routed from 90520738888109 2025-06-29 04:42:21,576 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-94ef03c7]) (logid:fc831c59) Seq 2-3926857400090362160: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:42:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-228f4040]) (logid:0dfaaaa2) No inactive management server node found 2025-06-29 04:42:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-228f4040]) (logid:0dfaaaa2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:22,718 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bd4082c3]) (logid:2824286e) HA health check task is running... 2025-06-29 04:42:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a502894]) (logid:421452a4) No inactive management server node found 2025-06-29 04:42:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a502894]) (logid:421452a4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eff555cf]) (logid:f9c0dce5) No inactive management server node found 2025-06-29 04:42:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eff555cf]) (logid:f9c0dce5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:25,381 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8729: Seq 7-8729: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:25,391 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8729 2025-06-29 04:42:26,719 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a959bdf4]) (logid:70224649) HA health check task is running... 2025-06-29 04:42:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52fa0c5d]) (logid:954272d6) No inactive management server node found 2025-06-29 04:42:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52fa0c5d]) (logid:954272d6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:27,266 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:42:27,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:42:27,271 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) Seq 2-4725683383995203858: Routed from 90520738888109 2025-06-29 04:42:27,324 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) Seq 1-4725683383995203858: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:42:27,357 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:42:27,358 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) Seq 2-3926857400090362161: Routed from 90520738888109 2025-06-29 04:42:27,451 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-644984d7]) (logid:e914420f) Seq 2-3926857400090362161: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:42:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e94e3aa5]) (logid:8786a136) Enabled non-edge zones available for scan: 1 2025-06-29 04:42:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e94e3aa5]) (logid:8786a136) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:42:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e94e3aa5]) (logid:8786a136) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:42:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-df134911]) (logid:51bb8538) Enabled non-edge zones available for scan: 1 2025-06-29 04:42:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-df134911]) (logid:51bb8538) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:42:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a736409f]) (logid:75efc74b) No inactive management server node found 2025-06-29 04:42:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a736409f]) (logid:75efc74b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:28,804 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) StorageCollector is running... 2025-06-29 04:42:28,812 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:42:28,812 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:42:28,812 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:42:28,813 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 2-3938116399158788169: Routed from 90520738888109 2025-06-29 04:42:28,868 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 4-3938116399158788169: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:42:28,872 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:42:28,872 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:42:28,872 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:42:28,873 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 2-4725683383995203859: Routed from 90520738888109 2025-06-29 04:42:28,920 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 1-4725683383995203859: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:42:28,931 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:42:28,931 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:42:28,931 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:42:28,932 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 2-4725683383995203860: Routed from 90520738888109 2025-06-29 04:42:29,020 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 1-4725683383995203860: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:42:29,033 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:42:29,033 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:42:29,033 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:42:29,034 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 2-4725683383995203861: Routed from 90520738888109 2025-06-29 04:42:29,119 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 1-4725683383995203861: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:42:29,120 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:42:29,120 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:42:29,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:42:29,121 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 2-3926857400090362162: Routed from 90520738888109 2025-06-29 04:42:29,125 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-1e9c3ed1]) (logid:dbda7e2f) Seq 2-3926857400090362162: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:42:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65d9cf8]) (logid:380335b1) No inactive management server node found 2025-06-29 04:42:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c65d9cf8]) (logid:380335b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:30,721 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-19ba4091]) (logid:791b2df3) HA health check task is running... 2025-06-29 04:42:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-392027c2]) (logid:26f2ba0b) No inactive management server node found 2025-06-29 04:42:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-392027c2]) (logid:26f2ba0b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:42:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-640a3cf8]) (logid:de1c5069) No inactive management server node found 2025-06-29 04:42:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-640a3cf8]) (logid:de1c5069) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-553aee37]) (logid:30a33ec8) No inactive management server node found 2025-06-29 04:42:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-553aee37]) (logid:30a33ec8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:34,722 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-0181c2be]) (logid:7a6d6c07) HA health check task is running... 2025-06-29 04:42:35,383 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8730: Seq 7-8730: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:35,393 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8730 2025-06-29 04:42:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-617a2484]) (logid:cce5e14f) No inactive management server node found 2025-06-29 04:42:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-617a2484]) (logid:cce5e14f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b30c8148]) (logid:78171035) No inactive management server node found 2025-06-29 04:42:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b30c8148]) (logid:78171035) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:38,724 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-fa77159a]) (logid:5f9bd121) HA health check task is running... 2025-06-29 04:42:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb098c1a]) (logid:4e5419dc) No inactive management server node found 2025-06-29 04:42:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb098c1a]) (logid:4e5419dc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-863ac6ea]) (logid:0a4d86ff) No inactive management server node found 2025-06-29 04:42:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-863ac6ea]) (logid:0a4d86ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e860448]) (logid:253ebf98) No inactive management server node found 2025-06-29 04:42:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e860448]) (logid:253ebf98) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:42,725 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-4becd69d]) (logid:1e66429f) HA health check task is running... 2025-06-29 04:42:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-e545ee3c]) (logid:8e4a4329) DbCollector is running... 2025-06-29 04:42:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-896f5d00]) (logid:37abdcb4) Found 0 routers to update status. 2025-06-29 04:42:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-896f5d00]) (logid:37abdcb4) Found 0 VPC's to update Redundant State. 2025-06-29 04:42:43,102 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-896f5d00]) (logid:37abdcb4) Found 0 networks to update RvR status. 2025-06-29 04:42:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-418d7782]) (logid:026a69db) No inactive management server node found 2025-06-29 04:42:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-418d7782]) (logid:026a69db) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) ManagementServerCollector is running... 2025-06-29 04:42:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) DB properties were already loaded 2025-06-29 04:42:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:42:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Execution of process [4067667] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:42:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Exit value of process [4067667] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:42:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Process [4067667] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:42:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) usage install: null 2025-06-29 04:42:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Sessions active: 0 2025-06-29 04:42:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:42:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067670] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:42:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:42:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067675] for command [/bin/bash -c uname -r ]. 2025-06-29 04:42:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:42:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067676] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:42:44,075 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) system memory from /proc: 4887515136 2025-06-29 04:42:44,075 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:42:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067681] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:42:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) free memory from /proc: 334823424 2025-06-29 04:42:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:42:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067686] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:42:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) used memory from /proc: 1019844 2025-06-29 04:42:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:42:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067687] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:42:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:42:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067688] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:42:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:42:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067689] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:42:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:42:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067696] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:42:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:42:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Successfully executed process [4067697] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail buffersdirect.count, 96 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail buffersdirect.used, 328650 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail buffersmapped.capacity, 0 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail buffersmapped.count, 0 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail buffersmapped.used, 0 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail jvmuptime, 4023832 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memoryheap.committed, 654311424 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memoryheap.init, 77594624 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memoryheap.usage, 0.20060847742614496 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorynon-heap.committed, 198639616 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:42:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorynon-heap.max, -1 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorynon-heap.usage, -1.94888056E8 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorynon-heap.used, 194888056 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11647414856038389 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21784916852734362 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388534843921661 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.Metaspace.usage, 0.9932693488605702 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.PS-Eden-Space.usage, 0.29181027470049986 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorytotal.committed, 852951040 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorytotal.init, 85262336 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorytotal.max, 1908932607 2025-06-29 04:42:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail memorytotal.used, 577836120 2025-06-29 04:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail threadsdeadlocks, [] 2025-06-29 04:42:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail threadsnew.count, 0 2025-06-29 04:42:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:42:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-474728ae]) (logid:aaf830eb) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":272065656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4023761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":577133824,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":382948064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334823424,"systemMemoryUsed":1019844,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.09,0.11],"systemCyclesUsage":[66420921,53840446,4590153274],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:42:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 133, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":272065656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4023761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":577133824,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":382948064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334823424,"systemMemoryUsed":1019844,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.09,0.11],"systemCyclesUsage":[66420921,53840446,4590153274],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:42:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:42:44,142 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 7 ms 2025-06-29 04:42:44,142 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-29:[ctx-aad16d1d]) (logid:441e7de8) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":272065656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4023761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":577133824,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":382948064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334823424,"systemMemoryUsed":1019844,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.09,0.11],"systemCyclesUsage":[66420921,53840446,4590153274],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:42:44,142 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 8ms. agent: 0, pdu seq: 133, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:42:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":272065656,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4023761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":577133824,"totalCommitted":852951040,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":382948064,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334823424,"systemMemoryUsed":1019844,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.09,0.11],"systemCyclesUsage":[66420921,53840446,4590153274],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:42:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b2ae516]) (logid:ba44d834) No inactive management server node found 2025-06-29 04:42:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b2ae516]) (logid:ba44d834) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:45,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8731: Seq 7-8731: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:45,392 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8731 2025-06-29 04:42:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:42:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:42:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a7f7e13]) (logid:e7046a1d) No inactive management server node found 2025-06-29 04:42:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a7f7e13]) (logid:e7046a1d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:42:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:42:46,726 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f07abd70]) (logid:f45ba285) HA health check task is running... 2025-06-29 04:42:47,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:47,078 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:42:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:42:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-149c57cf]) (logid:e6606647) No inactive management server node found 2025-06-29 04:42:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-149c57cf]) (logid:e6606647) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8d15aec8]) (logid:ae660bb1) Resetting hosts suitable for reconnect 2025-06-29 04:42:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8d15aec8]) (logid:ae660bb1) Completed resetting hosts suitable for reconnect 2025-06-29 04:42:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8d15aec8]) (logid:ae660bb1) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:42:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8d15aec8]) (logid:ae660bb1) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:42:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8d15aec8]) (logid:ae660bb1) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:42:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-8d15aec8]) (logid:ae660bb1) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:42:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-67375d51]) (logid:d9897778) VM scheduler.poll is being called at 2025-06-29T02:43:00+0000 2025-06-29 04:42:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-67375d51]) (logid:d9897778) Got 0 scheduled jobs to be executed at 2025-06-29T02:43:00+0000 2025-06-29 04:42:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-67375d51]) (logid:d9897778) Cleaned up 0 VM scheduled job entries 2025-06-29 04:42:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-b124ad47]) (logid:845518ff) VM Operation Thread Running 2025-06-29 04:42:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7cbd7f42]) (logid:5a63f3b1) ClusterDRS.poll is being called at 2025-06-29T02:43:00+0000 2025-06-29 04:42:48,357 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-7cbd7f42]) (logid:5a63f3b1) Removed 0 old drs migration plans 2025-06-29 04:42:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3aeee7a]) (logid:d3542b6e) No inactive management server node found 2025-06-29 04:42:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3aeee7a]) (logid:d3542b6e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:50,727 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a6bacb97]) (logid:76f88312) HA health check task is running... 2025-06-29 04:42:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dcf8b84]) (logid:6d8be278) No inactive management server node found 2025-06-29 04:42:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7dcf8b84]) (logid:6d8be278) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:52,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8696ea38]) (logid:52f0f451) No inactive management server node found 2025-06-29 04:42:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8696ea38]) (logid:52f0f451) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a219915]) (logid:debbded9) No inactive management server node found 2025-06-29 04:42:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a219915]) (logid:debbded9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:54,729 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0cde26ae]) (logid:385355aa) HA health check task is running... 2025-06-29 04:42:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7561ee05]) (logid:9e35fe2f) No inactive management server node found 2025-06-29 04:42:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7561ee05]) (logid:9e35fe2f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:55,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8733: Seq 7-8733: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:42:55,392 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8733 2025-06-29 04:42:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c8fc5c]) (logid:0c54772a) No inactive management server node found 2025-06-29 04:42:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-12c8fc5c]) (logid:0c54772a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-00b3a473]) (logid:6025b6b3) Enabled non-edge zones available for scan: 1 2025-06-29 04:42:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-00b3a473]) (logid:6025b6b3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:42:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-00b3a473]) (logid:6025b6b3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:42:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-60a64f39]) (logid:2101fe86) Enabled non-edge zones available for scan: 1 2025-06-29 04:42:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-60a64f39]) (logid:2101fe86) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:42:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e437588]) (logid:251eeb55) No inactive management server node found 2025-06-29 04:42:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e437588]) (logid:251eeb55) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:42:58,730 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-2cded569]) (logid:c29d1743) HA health check task is running... 2025-06-29 04:42:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21a5f439]) (logid:7e80fee7) No inactive management server node found 2025-06-29 04:42:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21a5f439]) (logid:7e80fee7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a88e77d]) (logid:43e4d5c5) No inactive management server node found 2025-06-29 04:43:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a88e77d]) (logid:43e4d5c5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:02,731 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-58fa5bd9]) (logid:d7df6c2a) HA health check task is running... 2025-06-29 04:43:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2d8bac]) (logid:82435de4) No inactive management server node found 2025-06-29 04:43:02,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc2d8bac]) (logid:82435de4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2e12fa4]) (logid:ebc6571a) No inactive management server node found 2025-06-29 04:43:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2e12fa4]) (logid:ebc6571a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:05,383 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8734: Seq 7-8734: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:05,393 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8734 2025-06-29 04:43:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f916c26]) (logid:1a59d30c) No inactive management server node found 2025-06-29 04:43:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f916c26]) (logid:1a59d30c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:06,733 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-86c854fe]) (logid:1d0644fe) HA health check task is running... 2025-06-29 04:43:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f3b99e2]) (logid:ff527c1a) No inactive management server node found 2025-06-29 04:43:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f3b99e2]) (logid:ff527c1a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c106c1ab]) (logid:294d187c) No inactive management server node found 2025-06-29 04:43:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c106c1ab]) (logid:294d187c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61b9b6b0]) (logid:cc17c5c3) No inactive management server node found 2025-06-29 04:43:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61b9b6b0]) (logid:cc17c5c3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:10,383 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8735: Seq 7-8735: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:10,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8735 2025-06-29 04:43:10,734 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-831cbe72]) (logid:433b9e3b) HA health check task is running... 2025-06-29 04:43:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e96dc646]) (logid:b4bc2473) No inactive management server node found 2025-06-29 04:43:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e96dc646]) (logid:b4bc2473) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dfb6cbae]) (logid:3aa2337b) Found 0 routers to update status. 2025-06-29 04:43:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dfb6cbae]) (logid:3aa2337b) Found 0 VPC's to update Redundant State. 2025-06-29 04:43:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-dfb6cbae]) (logid:3aa2337b) Found 0 networks to update RvR status. 2025-06-29 04:43:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1972d4c0]) (logid:599991a8) No inactive management server node found 2025-06-29 04:43:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1972d4c0]) (logid:599991a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:14,735 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-a7e01221]) (logid:563113a3) HA health check task is running... 2025-06-29 04:43:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2941d28]) (logid:e43a988f) No inactive management server node found 2025-06-29 04:43:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2941d28]) (logid:e43a988f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d420743]) (logid:489ad2e7) No inactive management server node found 2025-06-29 04:43:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d420743]) (logid:489ad2e7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa69c50e]) (logid:4e11f00e) No inactive management server node found 2025-06-29 04:43:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa69c50e]) (logid:4e11f00e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:18,072 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-683006b6]) (logid:7bbd441d) AutoScaling Monitor is running... 2025-06-29 04:43:18,146 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-f57ef038]) (logid:e671e4ba) Starting to remove all volume_stats rows older than [Sat Jun 28 16:43:18 CEST 2025]. 2025-06-29 04:43:18,146 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-f57ef038]) (logid:e671e4ba) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:43:18 CEST 2025]. 2025-06-29 04:43:18,736 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-52be731f]) (logid:73b85369) HA health check task is running... 2025-06-29 04:43:19,058 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-598a005e]) (logid:6faa2f05) Starting to remove all vm_stats rows older than [Sat Jun 28 16:43:19 CEST 2025]. 2025-06-29 04:43:19,081 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-598a005e]) (logid:6faa2f05) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:43:19 CEST 2025]. 2025-06-29 04:43:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cdbbbbb]) (logid:9fe6f157) No inactive management server node found 2025-06-29 04:43:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cdbbbbb]) (logid:9fe6f157) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:20,388 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8736: Seq 7-8736: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:20,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8736 2025-06-29 04:43:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-242d68b5]) (logid:7504e4d1) No inactive management server node found 2025-06-29 04:43:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-242d68b5]) (logid:7504e4d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:21,579 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:43:21,580 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:43:21,582 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) Seq 2-4725683383995203862: Routed from 90520738888109 2025-06-29 04:43:21,626 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) Seq 1-4725683383995203862: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:43:21,627 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:43:21,628 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) Seq 2-3926857400090362163: Routed from 90520738888109 2025-06-29 04:43:21,672 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-964e6a8a]) (logid:2129f0b0) Seq 2-3926857400090362163: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:43:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10974f30]) (logid:8ef0877e) No inactive management server node found 2025-06-29 04:43:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-10974f30]) (logid:8ef0877e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:22,737 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c703c769]) (logid:2f0c7313) HA health check task is running... 2025-06-29 04:43:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f9aeba5]) (logid:40736149) No inactive management server node found 2025-06-29 04:43:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f9aeba5]) (logid:40736149) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61d03e75]) (logid:2dbee32b) No inactive management server node found 2025-06-29 04:43:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61d03e75]) (logid:2dbee32b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:26,739 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-82142341]) (logid:68313f25) HA health check task is running... 2025-06-29 04:43:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73b1ea07]) (logid:d5e24c4d) No inactive management server node found 2025-06-29 04:43:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-73b1ea07]) (logid:d5e24c4d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:27,507 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:43:27,511 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:43:27,512 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) Seq 2-4725683383995203863: Routed from 90520738888109 2025-06-29 04:43:27,565 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) Seq 1-4725683383995203863: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:43:27,600 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:43:27,601 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) Seq 2-3926857400090362164: Routed from 90520738888109 2025-06-29 04:43:27,693 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-909c0d3c]) (logid:8b9364c2) Seq 2-3926857400090362164: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:43:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-571a29ea]) (logid:7cd8cd94) Enabled non-edge zones available for scan: 1 2025-06-29 04:43:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-571a29ea]) (logid:7cd8cd94) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:43:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-571a29ea]) (logid:7cd8cd94) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:43:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-bfb3e4d1]) (logid:7c78fc4c) Enabled non-edge zones available for scan: 1 2025-06-29 04:43:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-bfb3e4d1]) (logid:7c78fc4c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:43:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03d4d38c]) (logid:a5aabc97) No inactive management server node found 2025-06-29 04:43:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-03d4d38c]) (logid:a5aabc97) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:29,125 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) StorageCollector is running... 2025-06-29 04:43:29,132 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:43:29,132 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:43:29,132 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:43:29,133 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3938116399158788170: Routed from 90520738888109 2025-06-29 04:43:29,188 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 4-3938116399158788170: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:43:29,191 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:43:29,191 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:43:29,191 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:43:29,192 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3926857400090362165: Routed from 90520738888109 2025-06-29 04:43:29,205 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3926857400090362165: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:43:29,216 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:43:29,216 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:43:29,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:43:29,217 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3926857400090362166: Routed from 90520738888109 2025-06-29 04:43:29,310 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3926857400090362166: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:43:29,321 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:43:29,321 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:43:29,321 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:43:29,322 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-4725683383995203864: Routed from 90520738888109 2025-06-29 04:43:29,365 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 1-4725683383995203864: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:43:29,367 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:43:29,367 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:43:29,367 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:43:29,368 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3926857400090362167: Routed from 90520738888109 2025-06-29 04:43:29,412 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-72886447]) (logid:7c925f33) Seq 2-3926857400090362167: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:43:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d03d09f5]) (logid:3056a474) No inactive management server node found 2025-06-29 04:43:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d03d09f5]) (logid:3056a474) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:30,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8737: Seq 7-8737: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:30,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8737 2025-06-29 04:43:30,740 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-47784b9c]) (logid:df0070f8) HA health check task is running... 2025-06-29 04:43:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef4c3b79]) (logid:5c0282f9) No inactive management server node found 2025-06-29 04:43:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef4c3b79]) (logid:5c0282f9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c889212c]) (logid:ffec1807) No inactive management server node found 2025-06-29 04:43:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c889212c]) (logid:ffec1807) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09eb87e1]) (logid:04ef6974) No inactive management server node found 2025-06-29 04:43:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09eb87e1]) (logid:04ef6974) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:34,741 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-31ab5e21]) (logid:77d6c501) HA health check task is running... 2025-06-29 04:43:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48415bab]) (logid:0e41e969) No inactive management server node found 2025-06-29 04:43:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-48415bab]) (logid:0e41e969) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2685b26a]) (logid:2378f5b0) No inactive management server node found 2025-06-29 04:43:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2685b26a]) (logid:2378f5b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:38,742 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ba9c278d]) (logid:140b753a) HA health check task is running... 2025-06-29 04:43:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0ff4446]) (logid:5a6724e8) No inactive management server node found 2025-06-29 04:43:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c0ff4446]) (logid:5a6724e8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e82d9fd]) (logid:96ac19e2) No inactive management server node found 2025-06-29 04:43:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e82d9fd]) (logid:96ac19e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:40,403 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8738: Seq 7-8738: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:40,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8738 2025-06-29 04:43:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93cf778b]) (logid:dfd1e6eb) No inactive management server node found 2025-06-29 04:43:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93cf778b]) (logid:dfd1e6eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:42,744 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9ff0405a]) (logid:b751b2aa) HA health check task is running... 2025-06-29 04:43:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-be976dbf]) (logid:46189639) DbCollector is running... 2025-06-29 04:43:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-72e769b8]) (logid:87b077b4) Found 0 routers to update status. 2025-06-29 04:43:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-72e769b8]) (logid:87b077b4) Found 0 VPC's to update Redundant State. 2025-06-29 04:43:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-72e769b8]) (logid:87b077b4) Found 0 networks to update RvR status. 2025-06-29 04:43:43,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17cf2df5]) (logid:eb7a8166) No inactive management server node found 2025-06-29 04:43:43,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-17cf2df5]) (logid:eb7a8166) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) ManagementServerCollector is running... 2025-06-29 04:43:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) DB properties were already loaded 2025-06-29 04:43:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:43:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Execution of process [4067703] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:43:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Exit value of process [4067703] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:43:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Process [4067703] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:43:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) usage install: null 2025-06-29 04:43:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Sessions active: 0 2025-06-29 04:43:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:43:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067706] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:43:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:43:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067711] for command [/bin/bash -c uname -r ]. 2025-06-29 04:43:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:43:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067712] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:43:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) system memory from /proc: 4887515136 2025-06-29 04:43:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:43:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067717] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:43:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) free memory from /proc: 333148160 2025-06-29 04:43:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:43:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067722] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:43:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) used memory from /proc: 1019972 2025-06-29 04:43:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:43:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067723] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:43:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:43:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067724] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:43:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:43:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067725] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:43:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:43:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067732] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:43:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:43:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Successfully executed process [4067733] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail buffersdirect.count, 96 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail buffersdirect.used, 328650 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail buffersmapped.capacity, 0 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail buffersmapped.count, 0 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail buffersmapped.used, 0 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail jvmuptime, 4083829 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memoryheap.committed, 654311424 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memoryheap.init, 77594624 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memoryheap.usage, 0.21271234316931947 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorynon-heap.committed, 198705152 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorynon-heap.max, -1 2025-06-29 04:43:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorynon-heap.usage, -1.95017232E8 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorynon-heap.used, 195017232 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1167438683017862 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21845527710200954 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388534843921661 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.Metaspace.usage, 0.9929575392273979 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.PS-Eden-Space.usage, 0.3276415017225587 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorytotal.committed, 853016576 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorytotal.init, 85262336 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorytotal.max, 1908932607 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail memorytotal.used, 601281776 2025-06-29 04:43:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail threadsdeadlocks, [] 2025-06-29 04:43:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail threadsnew.count, 0 2025-06-29 04:43:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:43:44,130 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-724b03c3]) (logid:a5902279) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":248930576,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4083759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":600398080,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":406053528,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":333148160,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.13,0.13],"systemCyclesUsage":[66421252,53840750,4590176396],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:43:44,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 135, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":248930576,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4083759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":600398080,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":406053528,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":333148160,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.13,0.13],"systemCyclesUsage":[66421252,53840750,4590176396],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:43:44,131 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:43:44,136 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-29:[ctx-8bf62a73]) (logid:06f6ea90) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":248930576,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4083759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":600398080,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":406053528,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":333148160,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.13,0.13],"systemCyclesUsage":[66421252,53840750,4590176396],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:43:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:43:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 135, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:43:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":248930576,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4083759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":600398080,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":406053528,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":333148160,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.13,0.13],"systemCyclesUsage":[66421252,53840750,4590176396],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:43:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b40646c3]) (logid:3a0519de) No inactive management server node found 2025-06-29 04:43:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b40646c3]) (logid:3a0519de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:43:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:43:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f27087a8]) (logid:29edfea2) No inactive management server node found 2025-06-29 04:43:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f27087a8]) (logid:29edfea2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:43:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:43:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:43:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:43:46,745 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-6f14bcb3]) (logid:31bb7ea3) HA health check task is running... 2025-06-29 04:43:47,111 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:43:47,112 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:43:47,113 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:43:47,116 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:43:47,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc3cfaf3]) (logid:87b091a0) No inactive management server node found 2025-06-29 04:43:47,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc3cfaf3]) (logid:87b091a0) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:43:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f69df9e2]) (logid:8cb2a80f) VM scheduler.poll is being called at 2025-06-29T02:44:00+0000 2025-06-29 04:43:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f69df9e2]) (logid:8cb2a80f) Got 0 scheduled jobs to be executed at 2025-06-29T02:44:00+0000 2025-06-29 04:43:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f69df9e2]) (logid:8cb2a80f) Cleaned up 0 VM scheduled job entries 2025-06-29 04:43:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-551168f6]) (logid:a2deac11) VM Operation Thread Running 2025-06-29 04:43:48,351 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d5a7dc01]) (logid:febb643a) ClusterDRS.poll is being called at 2025-06-29T02:44:00+0000 2025-06-29 04:43:48,357 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d5a7dc01]) (logid:febb643a) Removed 0 old drs migration plans 2025-06-29 04:43:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e6331ff]) (logid:72b4a5a3) No inactive management server node found 2025-06-29 04:43:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e6331ff]) (logid:72b4a5a3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:50,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8740: Seq 7-8740: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:50,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8740 2025-06-29 04:43:50,747 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e5545ac8]) (logid:d728365f) HA health check task is running... 2025-06-29 04:43:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4087d064]) (logid:19ba5661) No inactive management server node found 2025-06-29 04:43:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4087d064]) (logid:19ba5661) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:52,369 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c51a4184]) (logid:4bc08a20) No inactive management server node found 2025-06-29 04:43:52,369 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c51a4184]) (logid:4bc08a20) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-590cec74]) (logid:e6bf3321) No inactive management server node found 2025-06-29 04:43:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-590cec74]) (logid:e6bf3321) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:54,748 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3a99cdd4]) (logid:fd34535e) HA health check task is running... 2025-06-29 04:43:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8814e617]) (logid:b311da38) No inactive management server node found 2025-06-29 04:43:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8814e617]) (logid:b311da38) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:55,385 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8741: Seq 7-8741: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:43:55,395 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8741 2025-06-29 04:43:56,862 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1e1e3cb]) (logid:40a3dd6b) No inactive management server node found 2025-06-29 04:43:56,862 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1e1e3cb]) (logid:40a3dd6b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8da0d586]) (logid:8891e875) Enabled non-edge zones available for scan: 1 2025-06-29 04:43:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8da0d586]) (logid:8891e875) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:43:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8da0d586]) (logid:8891e875) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:43:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d28ad126]) (logid:6ec59bca) Enabled non-edge zones available for scan: 1 2025-06-29 04:43:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d28ad126]) (logid:6ec59bca) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:43:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47b6b8dc]) (logid:c48d24f5) No inactive management server node found 2025-06-29 04:43:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47b6b8dc]) (logid:c48d24f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:43:58,749 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-daacdd1c]) (logid:34321a46) HA health check task is running... 2025-06-29 04:43:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2c92ece]) (logid:2422c73e) No inactive management server node found 2025-06-29 04:43:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2c92ece]) (logid:2422c73e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e79f248d]) (logid:cbd3a0f3) No inactive management server node found 2025-06-29 04:44:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e79f248d]) (logid:cbd3a0f3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:02,750 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b11c7b96]) (logid:34fdc7cd) HA health check task is running... 2025-06-29 04:44:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b16215ad]) (logid:45016df3) No inactive management server node found 2025-06-29 04:44:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b16215ad]) (logid:45016df3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efc57b0c]) (logid:37c1fee4) No inactive management server node found 2025-06-29 04:44:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-efc57b0c]) (logid:37c1fee4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:44:05,387 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8742: Seq 7-8742: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:44:05,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8742 2025-06-29 04:44:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0eeeed6f]) (logid:56d8f02c) No inactive management server node found 2025-06-29 04:44:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0eeeed6f]) (logid:56d8f02c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:06,752 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-58af7c7a]) (logid:2bda2a21) HA health check task is running... 2025-06-29 04:44:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2945e19b]) (logid:a6e189f5) No inactive management server node found 2025-06-29 04:44:07,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2945e19b]) (logid:a6e189f5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6f266f1]) (logid:6c9ed37a) No inactive management server node found 2025-06-29 04:44:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b6f266f1]) (logid:6c9ed37a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df7a9353]) (logid:82a54105) No inactive management server node found 2025-06-29 04:44:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df7a9353]) (logid:82a54105) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:10,753 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-871b59e6]) (logid:764e30ea) HA health check task is running... 2025-06-29 04:44:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-031b9c7d]) (logid:d302c05b) No inactive management server node found 2025-06-29 04:44:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-031b9c7d]) (logid:d302c05b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e3a6fa00]) (logid:5c15d581) Found 0 routers to update status. 2025-06-29 04:44:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e3a6fa00]) (logid:5c15d581) Found 0 VPC's to update Redundant State. 2025-06-29 04:44:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e3a6fa00]) (logid:5c15d581) Found 0 networks to update RvR status. 2025-06-29 04:44:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9073cf64]) (logid:86fd4d5b) No inactive management server node found 2025-06-29 04:44:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9073cf64]) (logid:86fd4d5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:14,755 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6182f50c]) (logid:46c6d708) HA health check task is running... 2025-06-29 04:44:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f80aaadf]) (logid:09f225d5) No inactive management server node found 2025-06-29 04:44:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f80aaadf]) (logid:09f225d5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:15,386 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8743: Seq 7-8743: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:44:15,396 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8743 2025-06-29 04:44:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bfcfa03]) (logid:c9b1b210) No inactive management server node found 2025-06-29 04:44:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bfcfa03]) (logid:c9b1b210) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-872ab6d3]) (logid:af00db4d) No inactive management server node found 2025-06-29 04:44:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-872ab6d3]) (logid:af00db4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:18,074 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-b737aa59]) (logid:84ca5572) AutoScaling Monitor is running... 2025-06-29 04:44:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-14550c6e]) (logid:708f4bb9) Resetting hosts suitable for reconnect 2025-06-29 04:44:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-14550c6e]) (logid:708f4bb9) Completed resetting hosts suitable for reconnect 2025-06-29 04:44:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-14550c6e]) (logid:708f4bb9) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:44:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-14550c6e]) (logid:708f4bb9) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:44:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-14550c6e]) (logid:708f4bb9) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:44:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-14550c6e]) (logid:708f4bb9) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:44:18,149 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-05671d35]) (logid:1c61a96f) Starting to remove all volume_stats rows older than [Sat Jun 28 16:44:18 CEST 2025]. 2025-06-29 04:44:18,149 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-05671d35]) (logid:1c61a96f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:44:18 CEST 2025]. 2025-06-29 04:44:18,756 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5fcb7aeb]) (logid:0eeb381c) HA health check task is running... 2025-06-29 04:44:19,083 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-41438c5f]) (logid:01986e52) Starting to remove all vm_stats rows older than [Sat Jun 28 16:44:19 CEST 2025]. 2025-06-29 04:44:19,106 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-41438c5f]) (logid:01986e52) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:44:19 CEST 2025]. 2025-06-29 04:44:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f214275]) (logid:68136ee5) No inactive management server node found 2025-06-29 04:44:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f214275]) (logid:68136ee5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26ecfc82]) (logid:e90e2bc6) No inactive management server node found 2025-06-29 04:44:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26ecfc82]) (logid:e90e2bc6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:21,675 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:44:21,677 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:44:21,678 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) Seq 2-4725683383995203865: Routed from 90520738888109 2025-06-29 04:44:21,721 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) Seq 1-4725683383995203865: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:44:21,722 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:44:21,723 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) Seq 2-3926857400090362168: Routed from 90520738888109 2025-06-29 04:44:21,768 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-19e6666f]) (logid:def2401c) Seq 2-3926857400090362168: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:44:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a88e9964]) (logid:9117ab69) No inactive management server node found 2025-06-29 04:44:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a88e9964]) (logid:9117ab69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:22,757 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-26cf9c83]) (logid:1b9ee7da) HA health check task is running... 2025-06-29 04:44:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66976982]) (logid:c9b0aeeb) No inactive management server node found 2025-06-29 04:44:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66976982]) (logid:c9b0aeeb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26b282a1]) (logid:5c182b2b) No inactive management server node found 2025-06-29 04:44:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-26b282a1]) (logid:5c182b2b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:25,388 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8744: Seq 7-8744: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:44:25,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8744 2025-06-29 04:44:26,758 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9ae40997]) (logid:b07ba648) HA health check task is running... 2025-06-29 04:44:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-944a04fb]) (logid:c2ea88c5) No inactive management server node found 2025-06-29 04:44:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-944a04fb]) (logid:c2ea88c5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:27,745 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:44:27,748 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:44:27,749 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) Seq 2-4725683383995203866: Routed from 90520738888109 2025-06-29 04:44:27,811 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) Seq 1-4725683383995203866: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:44:27,847 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:44:27,847 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) Seq 2-3926857400090362169: Routed from 90520738888109 2025-06-29 04:44:27,938 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-392332ff]) (logid:81108ec0) Seq 2-3926857400090362169: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:44:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e5f37468]) (logid:fa52503e) Enabled non-edge zones available for scan: 1 2025-06-29 04:44:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-e5f37468]) (logid:fa52503e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:44:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-e5f37468]) (logid:fa52503e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:44:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-99cfbaef]) (logid:f95c53e2) Enabled non-edge zones available for scan: 1 2025-06-29 04:44:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-99cfbaef]) (logid:f95c53e2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:44:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f35a092]) (logid:cdde438b) No inactive management server node found 2025-06-29 04:44:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f35a092]) (logid:cdde438b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:29,413 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) StorageCollector is running... 2025-06-29 04:44:29,418 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:44:29,418 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:44:29,418 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:44:29,419 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3938116399158788171: Routed from 90520738888109 2025-06-29 04:44:29,474 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 4-3938116399158788171: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:44:29,477 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:44:29,477 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:44:29,477 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:44:29,478 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3926857400090362170: Routed from 90520738888109 2025-06-29 04:44:29,491 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3926857400090362170: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:44:29,503 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:44:29,503 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:44:29,503 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:44:29,504 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3926857400090362171: Routed from 90520738888109 2025-06-29 04:44:29,597 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3926857400090362171: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:44:29,608 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:44:29,608 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:44:29,608 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:44:29,609 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-4725683383995203867: Routed from 90520738888109 2025-06-29 04:44:29,654 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 1-4725683383995203867: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:44:29,656 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:44:29,656 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:44:29,657 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:44:29,658 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3926857400090362172: Routed from 90520738888109 2025-06-29 04:44:29,701 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-f3f4ed91]) (logid:5fcf24a4) Seq 2-3926857400090362172: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:44:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36dae71e]) (logid:8460448f) No inactive management server node found 2025-06-29 04:44:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-36dae71e]) (logid:8460448f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:30,760 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2d04652f]) (logid:e659c934) HA health check task is running... 2025-06-29 04:44:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76431c81]) (logid:7be266c5) No inactive management server node found 2025-06-29 04:44:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76431c81]) (logid:7be266c5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8320d1bf]) (logid:80d3a272) No inactive management server node found 2025-06-29 04:44:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8320d1bf]) (logid:80d3a272) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f055264b]) (logid:11c6cae9) No inactive management server node found 2025-06-29 04:44:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f055264b]) (logid:11c6cae9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:34,761 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-63634d33]) (logid:5383cbdf) HA health check task is running... 2025-06-29 04:44:35,391 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8745: Seq 7-8745: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:44:35,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8745 2025-06-29 04:44:35,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6507911]) (logid:8a1e77fd) No inactive management server node found 2025-06-29 04:44:35,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f6507911]) (logid:8a1e77fd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-802f7611]) (logid:faf674ad) No inactive management server node found 2025-06-29 04:44:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-802f7611]) (logid:faf674ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:38,762 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7b54f78a]) (logid:b7af7d4b) HA health check task is running... 2025-06-29 04:44:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef67c538]) (logid:5082d948) No inactive management server node found 2025-06-29 04:44:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ef67c538]) (logid:5082d948) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc40cc9f]) (logid:4b264985) No inactive management server node found 2025-06-29 04:44:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc40cc9f]) (logid:4b264985) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a437ba1]) (logid:5b090d32) No inactive management server node found 2025-06-29 04:44:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a437ba1]) (logid:5b090d32) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:42,763 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3763facb]) (logid:b3664cb9) HA health check task is running... 2025-06-29 04:44:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-3ec66f05]) (logid:cd1ca3b2) DbCollector is running... 2025-06-29 04:44:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-09333563]) (logid:b9a34bed) Found 0 routers to update status. 2025-06-29 04:44:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-09333563]) (logid:b9a34bed) Found 0 VPC's to update Redundant State. 2025-06-29 04:44:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-09333563]) (logid:b9a34bed) Found 0 networks to update RvR status. 2025-06-29 04:44:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb2e8cf0]) (logid:bf9d69d9) No inactive management server node found 2025-06-29 04:44:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bb2e8cf0]) (logid:bf9d69d9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) ManagementServerCollector is running... 2025-06-29 04:44:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) DB properties were already loaded 2025-06-29 04:44:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:44:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Execution of process [4067746] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:44:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Exit value of process [4067746] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:44:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Process [4067746] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:44:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) usage install: null 2025-06-29 04:44:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Sessions active: 0 2025-06-29 04:44:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:44:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067749] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:44:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:44:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067754] for command [/bin/bash -c uname -r ]. 2025-06-29 04:44:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:44:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067755] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:44:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) system memory from /proc: 4887515136 2025-06-29 04:44:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:44:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067760] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:44:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) free memory from /proc: 331825152 2025-06-29 04:44:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:44:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067765] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:44:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) used memory from /proc: 1019972 2025-06-29 04:44:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:44:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067766] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:44:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:44:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067767] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:44:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:44:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067768] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:44:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:44:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067775] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:44:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:44:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Successfully executed process [4067776] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail buffersdirect.count, 96 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail buffersdirect.used, 328650 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail buffersmapped.capacity, 0 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail buffersmapped.count, 0 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail buffersmapped.used, 0 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail jvmuptime, 4143832 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memoryheap.committed, 654311424 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memoryheap.init, 77594624 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memoryheap.usage, 0.22401654108053248 2025-06-29 04:44:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorynon-heap.committed, 198705152 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorynon-heap.max, -1 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorynon-heap.usage, -1.95148048E8 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorynon-heap.used, 195148048 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11713855471874167 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21898744459626088 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388534843921661 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.Metaspace.usage, 0.9930784644150152 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.PS-Eden-Space.usage, 0.3607522501857121 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorytotal.committed, 853016576 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorytotal.init, 85262336 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorytotal.max, 1908932607 2025-06-29 04:44:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail memorytotal.used, 622952616 2025-06-29 04:44:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail threadsdeadlocks, [] 2025-06-29 04:44:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail threadsnew.count, 0 2025-06-29 04:44:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:44:44,135 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-38ee6b7e]) (logid:af414be8) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":227415712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":622043760,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":427632480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":331825152,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.12,0.12],"systemCyclesUsage":[66421612,53841062,4590199477],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:44:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 137, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":227415712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":622043760,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":427632480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":331825152,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.12,0.12],"systemCyclesUsage":[66421612,53841062,4590199477],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:44:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:44:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 04:44:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 4ms. agent: 0, pdu seq: 137, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":227415712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":622043760,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":427632480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":331825152,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.12,0.12],"systemCyclesUsage":[66421612,53841062,4590199477],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:44:44,141 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-30:[ctx-dc95dcb9]) (logid:b10adf08) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:44:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":227415712,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4143759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.14,"totalInit":85262336,"totalUsed":622043760,"totalCommitted":853016576,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":427632480,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":331825152,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.14,0.12,0.12],"systemCyclesUsage":[66421612,53841062,4590199477],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:44:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40e781dc]) (logid:cae1f57f) No inactive management server node found 2025-06-29 04:44:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40e781dc]) (logid:cae1f57f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:45,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8746: Seq 7-8746: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:44:45,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8746 2025-06-29 04:44:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:44:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:44:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d11a30f9]) (logid:19f82b7d) No inactive management server node found 2025-06-29 04:44:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d11a30f9]) (logid:19f82b7d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:46,646 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:44:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:44:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:44:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:44:46,765 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-1aadf841]) (logid:d578160f) HA health check task is running... 2025-06-29 04:44:47,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:44:47,102 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:44:47,104 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:44:47,106 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:44:47,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81da2ae4]) (logid:aaebc1ec) No inactive management server node found 2025-06-29 04:44:47,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-81da2ae4]) (logid:aaebc1ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-757fe577]) (logid:56648d66) VM scheduler.poll is being called at 2025-06-29T02:45:00+0000 2025-06-29 04:44:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-757fe577]) (logid:56648d66) Got 0 scheduled jobs to be executed at 2025-06-29T02:45:00+0000 2025-06-29 04:44:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-757fe577]) (logid:56648d66) Cleaned up 0 VM scheduled job entries 2025-06-29 04:44:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-98f10b62]) (logid:7b62f2a9) VM Operation Thread Running 2025-06-29 04:44:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d58b342c]) (logid:b0e79ea9) ClusterDRS.poll is being called at 2025-06-29T02:45:00+0000 2025-06-29 04:44:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-d58b342c]) (logid:b0e79ea9) Removed 0 old drs migration plans 2025-06-29 04:44:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b4cd094]) (logid:dff42a49) No inactive management server node found 2025-06-29 04:44:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7b4cd094]) (logid:dff42a49) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:50,766 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-de2dd339]) (logid:70b9ba36) HA health check task is running... 2025-06-29 04:44:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2a69870]) (logid:c1612714) No inactive management server node found 2025-06-29 04:44:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2a69870]) (logid:c1612714) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3078e860]) (logid:2635771e) No inactive management server node found 2025-06-29 04:44:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3078e860]) (logid:2635771e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c43594f]) (logid:342fb3d9) No inactive management server node found 2025-06-29 04:44:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c43594f]) (logid:342fb3d9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:54,767 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-9d3c75d3]) (logid:84c15528) HA health check task is running... 2025-06-29 04:44:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45d83947]) (logid:5695025f) No inactive management server node found 2025-06-29 04:44:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45d83947]) (logid:5695025f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:55,388 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8748: Seq 7-8748: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:44:55,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8748 2025-06-29 04:44:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7705b27a]) (logid:b1b9f43e) No inactive management server node found 2025-06-29 04:44:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7705b27a]) (logid:b1b9f43e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8bd937e9]) (logid:63b64163) Enabled non-edge zones available for scan: 1 2025-06-29 04:44:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8bd937e9]) (logid:63b64163) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:44:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8bd937e9]) (logid:63b64163) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:44:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ff8a224d]) (logid:6873ab29) Enabled non-edge zones available for scan: 1 2025-06-29 04:44:58,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ff8a224d]) (logid:6873ab29) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:44:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb969558]) (logid:d519b103) No inactive management server node found 2025-06-29 04:44:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb969558]) (logid:d519b103) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:44:58,768 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2012d249]) (logid:805d045f) HA health check task is running... 2025-06-29 04:44:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f157345]) (logid:b562726c) No inactive management server node found 2025-06-29 04:44:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4f157345]) (logid:b562726c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9817ebcd]) (logid:edd027f1) No inactive management server node found 2025-06-29 04:45:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9817ebcd]) (logid:edd027f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:02,770 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-323634b5]) (logid:9ff7d452) HA health check task is running... 2025-06-29 04:45:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b987d17f]) (logid:f761d91d) No inactive management server node found 2025-06-29 04:45:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b987d17f]) (logid:f761d91d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab81914b]) (logid:f67d5634) No inactive management server node found 2025-06-29 04:45:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab81914b]) (logid:f67d5634) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:05,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8749: Seq 7-8749: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:45:05,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8749 2025-06-29 04:45:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0663fe71]) (logid:2bf96115) No inactive management server node found 2025-06-29 04:45:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0663fe71]) (logid:2bf96115) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:06,771 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4b43887b]) (logid:d1c0eac0) HA health check task is running... 2025-06-29 04:45:07,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bf6e0be]) (logid:9d880955) No inactive management server node found 2025-06-29 04:45:07,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bf6e0be]) (logid:9d880955) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a42ee9f]) (logid:a20cd8b9) No inactive management server node found 2025-06-29 04:45:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a42ee9f]) (logid:a20cd8b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4469bb6]) (logid:7c5bf547) No inactive management server node found 2025-06-29 04:45:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4469bb6]) (logid:7c5bf547) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:10,772 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-5421e54d]) (logid:1cb03c8e) HA health check task is running... 2025-06-29 04:45:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07564b7a]) (logid:92eb0b58) No inactive management server node found 2025-06-29 04:45:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-07564b7a]) (logid:92eb0b58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a9a1bbcd]) (logid:c017bfd1) Found 0 routers to update status. 2025-06-29 04:45:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a9a1bbcd]) (logid:c017bfd1) Found 0 VPC's to update Redundant State. 2025-06-29 04:45:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a9a1bbcd]) (logid:c017bfd1) Found 0 networks to update RvR status. 2025-06-29 04:45:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d27d7a8]) (logid:2e978c83) No inactive management server node found 2025-06-29 04:45:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d27d7a8]) (logid:2e978c83) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:14,774 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7671214d]) (logid:ef8692dc) HA health check task is running... 2025-06-29 04:45:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d499250]) (logid:d4e634f7) No inactive management server node found 2025-06-29 04:45:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d499250]) (logid:d4e634f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:15,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8750: Seq 7-8750: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:45:15,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8750 2025-06-29 04:45:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b32fbda]) (logid:5e0538d4) No inactive management server node found 2025-06-29 04:45:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b32fbda]) (logid:5e0538d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8df82dd3]) (logid:aaa32e3d) No inactive management server node found 2025-06-29 04:45:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8df82dd3]) (logid:aaa32e3d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:18,076 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-fe95faee]) (logid:c9febbb1) AutoScaling Monitor is running... 2025-06-29 04:45:18,151 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-165a1418]) (logid:3f73bf88) Starting to remove all volume_stats rows older than [Sat Jun 28 16:45:18 CEST 2025]. 2025-06-29 04:45:18,152 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-165a1418]) (logid:3f73bf88) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:45:18 CEST 2025]. 2025-06-29 04:45:18,775 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e0287f78]) (logid:471680c1) HA health check task is running... 2025-06-29 04:45:19,107 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-e23a4136]) (logid:4acac9d6) Starting to remove all vm_stats rows older than [Sat Jun 28 16:45:19 CEST 2025]. 2025-06-29 04:45:19,131 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-e23a4136]) (logid:4acac9d6) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:45:19 CEST 2025]. 2025-06-29 04:45:19,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b56d618]) (logid:85b8cb1a) No inactive management server node found 2025-06-29 04:45:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b56d618]) (logid:85b8cb1a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:20,389 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8751: Seq 7-8751: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:45:20,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8751 2025-06-29 04:45:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2f2da97]) (logid:4259bcd8) No inactive management server node found 2025-06-29 04:45:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a2f2da97]) (logid:4259bcd8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:21,771 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:45:21,772 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:45:21,775 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) Seq 2-4725683383995203868: Routed from 90520738888109 2025-06-29 04:45:21,818 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) Seq 1-4725683383995203868: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:45:21,820 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:45:21,821 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) Seq 2-3926857400090362173: Routed from 90520738888109 2025-06-29 04:45:21,864 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-77a118c5]) (logid:ce7c1aae) Seq 2-3926857400090362173: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:45:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c17348cd]) (logid:4dcfc91c) No inactive management server node found 2025-06-29 04:45:22,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c17348cd]) (logid:4dcfc91c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:22,776 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-2836a554]) (logid:36467176) HA health check task is running... 2025-06-29 04:45:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0140d42]) (logid:5f447cc2) No inactive management server node found 2025-06-29 04:45:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0140d42]) (logid:5f447cc2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f8af654]) (logid:aa4c9a12) No inactive management server node found 2025-06-29 04:45:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f8af654]) (logid:aa4c9a12) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:26,777 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-13798418]) (logid:89d24a6f) HA health check task is running... 2025-06-29 04:45:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66d84111]) (logid:91d4c3ab) No inactive management server node found 2025-06-29 04:45:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-66d84111]) (logid:91d4c3ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:27,992 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:45:27,995 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:45:27,996 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) Seq 2-4725683383995203869: Routed from 90520738888109 2025-06-29 04:45:28,054 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) Seq 1-4725683383995203869: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:45:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0b12d631]) (logid:9d525cf9) Enabled non-edge zones available for scan: 1 2025-06-29 04:45:28,090 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:45:28,091 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) Seq 2-3926857400090362174: Routed from 90520738888109 2025-06-29 04:45:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0b12d631]) (logid:9d525cf9) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:45:28,098 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0b12d631]) (logid:9d525cf9) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:45:28,187 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-361f3bd5]) (logid:361e5779) Seq 2-3926857400090362174: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:45:28,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4dab370e]) (logid:788eaef0) Enabled non-edge zones available for scan: 1 2025-06-29 04:45:28,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-4dab370e]) (logid:788eaef0) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:45:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f847460c]) (logid:6af7e006) No inactive management server node found 2025-06-29 04:45:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f847460c]) (logid:6af7e006) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:29,702 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) StorageCollector is running... 2025-06-29 04:45:29,708 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:45:29,708 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:45:29,708 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:45:29,709 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 2-3938116399158788172: Routed from 90520738888109 2025-06-29 04:45:29,763 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 4-3938116399158788172: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:45:29,766 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:45:29,766 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:45:29,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:45:29,768 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 2-4725683383995203870: Routed from 90520738888109 2025-06-29 04:45:29,816 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 1-4725683383995203870: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:45:29,826 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:45:29,826 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:45:29,826 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:45:29,827 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 2-4725683383995203871: Routed from 90520738888109 2025-06-29 04:45:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60b021c4]) (logid:ffee7af6) No inactive management server node found 2025-06-29 04:45:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60b021c4]) (logid:ffee7af6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:29,915 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 1-4725683383995203871: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:45:29,926 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:45:29,926 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:45:29,926 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:45:29,927 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 2-4725683383995203872: Routed from 90520738888109 2025-06-29 04:45:30,012 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 1-4725683383995203872: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:45:30,013 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:45:30,014 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:45:30,014 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:45:30,014 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 2-3926857400090362175: Routed from 90520738888109 2025-06-29 04:45:30,024 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-5f9541ee]) (logid:5b5cce64) Seq 2-3926857400090362175: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:45:30,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8752: Seq 7-8752: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:45:30,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8752 2025-06-29 04:45:30,779 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1f3b1b48]) (logid:1f3ec0fa) HA health check task is running... 2025-06-29 04:45:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fbcdd92]) (logid:cce60d6c) No inactive management server node found 2025-06-29 04:45:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8fbcdd92]) (logid:cce60d6c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b4690b]) (logid:51efbf85) No inactive management server node found 2025-06-29 04:45:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0b4690b]) (logid:51efbf85) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7110c9a4]) (logid:8e28f0ec) No inactive management server node found 2025-06-29 04:45:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7110c9a4]) (logid:8e28f0ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:45:34,780 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3aa11e6e]) (logid:793cdddd) HA health check task is running... 2025-06-29 04:45:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89252db6]) (logid:46fe92eb) No inactive management server node found 2025-06-29 04:45:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89252db6]) (logid:46fe92eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b82e58fe]) (logid:863d3c7a) No inactive management server node found 2025-06-29 04:45:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b82e58fe]) (logid:863d3c7a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:38,781 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-838664b3]) (logid:34bffaa7) HA health check task is running... 2025-06-29 04:45:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8568a627]) (logid:4412ca0d) No inactive management server node found 2025-06-29 04:45:38,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8568a627]) (logid:4412ca0d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-138d0a4d]) (logid:e93b6e8a) No inactive management server node found 2025-06-29 04:45:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-138d0a4d]) (logid:e93b6e8a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:40,390 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8753: Seq 7-8753: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:45:40,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8753 2025-06-29 04:45:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c72b09f]) (logid:94f66505) No inactive management server node found 2025-06-29 04:45:41,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c72b09f]) (logid:94f66505) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:42,782 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d8cd3685]) (logid:9ae99d62) HA health check task is running... 2025-06-29 04:45:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-0effd48b]) (logid:f4befc89) DbCollector is running... 2025-06-29 04:45:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0894757c]) (logid:9aa3188c) Found 0 routers to update status. 2025-06-29 04:45:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0894757c]) (logid:9aa3188c) Found 0 VPC's to update Redundant State. 2025-06-29 04:45:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-0894757c]) (logid:9aa3188c) Found 0 networks to update RvR status. 2025-06-29 04:45:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5af1c83]) (logid:4d11d36e) No inactive management server node found 2025-06-29 04:45:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e5af1c83]) (logid:4d11d36e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) ManagementServerCollector is running... 2025-06-29 04:45:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) DB properties were already loaded 2025-06-29 04:45:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:45:44,056 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Execution of process [4067802] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:45:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Exit value of process [4067802] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:45:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Process [4067802] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:45:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) usage install: null 2025-06-29 04:45:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Sessions active: 0 2025-06-29 04:45:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:45:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067805] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:45:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:45:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067810] for command [/bin/bash -c uname -r ]. 2025-06-29 04:45:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:45:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067811] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:45:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) system memory from /proc: 4887515136 2025-06-29 04:45:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:45:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067816] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:45:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) free memory from /proc: 334180352 2025-06-29 04:45:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:45:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067821] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:45:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) used memory from /proc: 1019972 2025-06-29 04:45:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:45:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067822] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:45:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:45:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067823] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:45:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:45:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067824] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:45:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:45:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067831] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:45:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:45:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Successfully executed process [4067832] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:45:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:45:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail buffersdirect.count, 96 2025-06-29 04:45:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail buffersdirect.used, 328650 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail buffersmapped.capacity, 0 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail buffersmapped.count, 0 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail buffersmapped.used, 0 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail jvmuptime, 4203833 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memoryheap.committed, 654311424 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memoryheap.init, 77594624 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memoryheap.usage, 0.23654057042541754 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorynon-heap.committed, 198770688 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorynon-heap.max, -1 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorynon-heap.usage, -1.95258984E8 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorynon-heap.used, 195258984 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11751137196747534 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21940609691072083 2025-06-29 04:45:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.Compressed-Class-Space.usage, 0.012388534843921661 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.Metaspace.usage, 0.9927105767258556 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.PS-Eden-Space.usage, 0.39754588751904335 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorytotal.committed, 853082112 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorytotal.init, 85262336 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorytotal.max, 1908932607 2025-06-29 04:45:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail memorytotal.used, 647003016 2025-06-29 04:45:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail threadsdeadlocks, [] 2025-06-29 04:45:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail threadsnew.count, 0 2025-06-29 04:45:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) not storing detail threadstimed_waiting.count, 83 2025-06-29 04:45:44,135 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-254f42b7]) (logid:4f6efc89) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":203416288,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":646154120,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":451540008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334180352,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.11,0.11],"systemCyclesUsage":[66421937,53841350,4590222617],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:45:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 139, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":203416288,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":646154120,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":451540008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334180352,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.11,0.11],"systemCyclesUsage":[66421937,53841350,4590222617],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:45:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:45:44,141 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-30:[ctx-0c5b38d1]) (logid:ed0a3b7b) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":203416288,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":646154120,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":451540008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334180352,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.11,0.11],"systemCyclesUsage":[66421937,53841350,4590222617],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:45:44,141 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:45:44,141 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 139, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:45:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":203416288,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4203760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.11,"totalInit":85262336,"totalUsed":646154120,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":451540008,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":202,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":334180352,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.11,0.11,0.11],"systemCyclesUsage":[66421937,53841350,4590222617],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:45:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2c5110e]) (logid:b18479e8) No inactive management server node found 2025-06-29 04:45:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2c5110e]) (logid:b18479e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:45:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:45:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af9f3b0a]) (logid:aacc6e9b) No inactive management server node found 2025-06-29 04:45:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af9f3b0a]) (logid:aacc6e9b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:45:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:45:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:45:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:45:46,783 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-d47c1ce0]) (logid:fd6ed72d) HA health check task is running... 2025-06-29 04:45:47,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:45:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:45:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:45:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:45:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0833a92e]) (logid:f604efa9) No inactive management server node found 2025-06-29 04:45:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0833a92e]) (logid:f604efa9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:48,094 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-28b89236]) (logid:8a5bf404) Resetting hosts suitable for reconnect 2025-06-29 04:45:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-28b89236]) (logid:8a5bf404) Completed resetting hosts suitable for reconnect 2025-06-29 04:45:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-28b89236]) (logid:8a5bf404) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:45:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-28b89236]) (logid:8a5bf404) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:45:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-28b89236]) (logid:8a5bf404) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:45:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-28b89236]) (logid:8a5bf404) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:45:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9a86ab1c]) (logid:e0f93a4d) VM scheduler.poll is being called at 2025-06-29T02:46:00+0000 2025-06-29 04:45:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9a86ab1c]) (logid:e0f93a4d) Got 0 scheduled jobs to be executed at 2025-06-29T02:46:00+0000 2025-06-29 04:45:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-9a86ab1c]) (logid:e0f93a4d) Cleaned up 0 VM scheduled job entries 2025-06-29 04:45:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-52651703]) (logid:b4003f0e) VM Operation Thread Running 2025-06-29 04:45:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ab1fe364]) (logid:bae8966e) ClusterDRS.poll is being called at 2025-06-29T02:46:00+0000 2025-06-29 04:45:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ab1fe364]) (logid:bae8966e) Removed 0 old drs migration plans 2025-06-29 04:45:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af890eb5]) (logid:88e69e10) No inactive management server node found 2025-06-29 04:45:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-af890eb5]) (logid:88e69e10) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:50,392 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8755: Seq 7-8755: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:45:50,406 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8755 2025-06-29 04:45:50,784 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c0ce031f]) (logid:0c890d12) HA health check task is running... 2025-06-29 04:45:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3362fb21]) (logid:53730f27) No inactive management server node found 2025-06-29 04:45:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3362fb21]) (logid:53730f27) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7974bfaa]) (logid:4004a72b) No inactive management server node found 2025-06-29 04:45:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7974bfaa]) (logid:4004a72b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-408c7425]) (logid:66b2fdca) No inactive management server node found 2025-06-29 04:45:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-408c7425]) (logid:66b2fdca) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:54,786 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7076169a]) (logid:1b90f058) HA health check task is running... 2025-06-29 04:45:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb564529]) (logid:81dc1b2f) No inactive management server node found 2025-06-29 04:45:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb564529]) (logid:81dc1b2f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c62f1c0]) (logid:d1aad4ee) No inactive management server node found 2025-06-29 04:45:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c62f1c0]) (logid:d1aad4ee) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8fb759e7]) (logid:587edfd7) Enabled non-edge zones available for scan: 1 2025-06-29 04:45:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-8fb759e7]) (logid:587edfd7) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:45:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8fb759e7]) (logid:587edfd7) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:45:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b69201f1]) (logid:c68a122f) Enabled non-edge zones available for scan: 1 2025-06-29 04:45:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b69201f1]) (logid:c68a122f) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:45:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40125d42]) (logid:0fe139fc) No inactive management server node found 2025-06-29 04:45:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-40125d42]) (logid:0fe139fc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:45:58,787 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-34b2092c]) (logid:a4245e81) HA health check task is running... 2025-06-29 04:45:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e41286f6]) (logid:9e21b332) No inactive management server node found 2025-06-29 04:45:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e41286f6]) (logid:9e21b332) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:00,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8756: Seq 7-8756: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:00,408 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8756 2025-06-29 04:46:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30bd1567]) (logid:ad65f3c2) No inactive management server node found 2025-06-29 04:46:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30bd1567]) (logid:ad65f3c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:02,788 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4353c9bf]) (logid:7fc1a27c) HA health check task is running... 2025-06-29 04:46:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4946d94]) (logid:b803fb59) No inactive management server node found 2025-06-29 04:46:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f4946d94]) (logid:b803fb59) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fd850d2]) (logid:33212acb) No inactive management server node found 2025-06-29 04:46:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fd850d2]) (logid:33212acb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acc8fae5]) (logid:71403716) No inactive management server node found 2025-06-29 04:46:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-acc8fae5]) (logid:71403716) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:06,790 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-ef9c2ab8]) (logid:196b2cb8) HA health check task is running... 2025-06-29 04:46:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-604f6fcb]) (logid:2a376243) No inactive management server node found 2025-06-29 04:46:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-604f6fcb]) (logid:2a376243) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eff7a1ae]) (logid:79a00628) No inactive management server node found 2025-06-29 04:46:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eff7a1ae]) (logid:79a00628) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:46:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6233a059]) (logid:fc188d89) No inactive management server node found 2025-06-29 04:46:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6233a059]) (logid:fc188d89) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:10,392 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8757: Seq 7-8757: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:10,402 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8757 2025-06-29 04:46:10,791 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-a28299af]) (logid:e3ef0411) HA health check task is running... 2025-06-29 04:46:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adcd92ca]) (logid:0d600f7d) No inactive management server node found 2025-06-29 04:46:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adcd92ca]) (logid:0d600f7d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8579f636]) (logid:9b00950a) Found 0 routers to update status. 2025-06-29 04:46:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8579f636]) (logid:9b00950a) Found 0 VPC's to update Redundant State. 2025-06-29 04:46:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8579f636]) (logid:9b00950a) Found 0 networks to update RvR status. 2025-06-29 04:46:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6921532a]) (logid:aca66f69) No inactive management server node found 2025-06-29 04:46:13,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6921532a]) (logid:aca66f69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:14,793 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-0376532b]) (logid:f328bdb5) HA health check task is running... 2025-06-29 04:46:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-073d832f]) (logid:3526a403) No inactive management server node found 2025-06-29 04:46:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-073d832f]) (logid:3526a403) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b29f56e]) (logid:6c297725) No inactive management server node found 2025-06-29 04:46:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b29f56e]) (logid:6c297725) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9908778]) (logid:866c5e72) No inactive management server node found 2025-06-29 04:46:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a9908778]) (logid:866c5e72) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:18,078 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-3fe6fdda]) (logid:dc8c7cea) AutoScaling Monitor is running... 2025-06-29 04:46:18,155 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-9847c18f]) (logid:4770f639) Starting to remove all volume_stats rows older than [Sat Jun 28 16:46:18 CEST 2025]. 2025-06-29 04:46:18,155 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-9847c18f]) (logid:4770f639) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:46:18 CEST 2025]. 2025-06-29 04:46:18,794 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-4040b614]) (logid:55c265ea) HA health check task is running... 2025-06-29 04:46:19,132 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a56265ec]) (logid:9f6272a1) Starting to remove all vm_stats rows older than [Sat Jun 28 16:46:19 CEST 2025]. 2025-06-29 04:46:19,159 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a56265ec]) (logid:9f6272a1) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:46:19 CEST 2025]. 2025-06-29 04:46:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2053c2fb]) (logid:5ec6ffbb) No inactive management server node found 2025-06-29 04:46:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2053c2fb]) (logid:5ec6ffbb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:20,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8758: Seq 7-8758: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:20,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8758 2025-06-29 04:46:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a193b41c]) (logid:a17e13c2) No inactive management server node found 2025-06-29 04:46:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a193b41c]) (logid:a17e13c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:21,867 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:46:21,869 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:46:21,870 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) Seq 2-4725683383995203873: Routed from 90520738888109 2025-06-29 04:46:21,913 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) Seq 1-4725683383995203873: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:46:21,915 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:46:21,915 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) Seq 2-3926857400090362176: Routed from 90520738888109 2025-06-29 04:46:21,960 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47355f64]) (logid:90af7123) Seq 2-3926857400090362176: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:46:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3b67478]) (logid:6a00cf43) No inactive management server node found 2025-06-29 04:46:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3b67478]) (logid:6a00cf43) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:22,795 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d5ff0191]) (logid:cc97d80f) HA health check task is running... 2025-06-29 04:46:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87af4ec5]) (logid:84d4545a) No inactive management server node found 2025-06-29 04:46:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87af4ec5]) (logid:84d4545a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82462dce]) (logid:dcc5e89d) No inactive management server node found 2025-06-29 04:46:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-82462dce]) (logid:dcc5e89d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:26,796 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-42ad5578]) (logid:fb1fdca7) HA health check task is running... 2025-06-29 04:46:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0458fecb]) (logid:1bcf1c53) No inactive management server node found 2025-06-29 04:46:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0458fecb]) (logid:1bcf1c53) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5f74b759]) (logid:05a6c8f8) Enabled non-edge zones available for scan: 1 2025-06-29 04:46:28,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5f74b759]) (logid:05a6c8f8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:46:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5f74b759]) (logid:05a6c8f8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:46:28,246 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:46:28,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:46:28,250 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) Seq 2-4725683383995203874: Routed from 90520738888109 2025-06-29 04:46:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7ab1134d]) (logid:e76a82b2) Enabled non-edge zones available for scan: 1 2025-06-29 04:46:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7ab1134d]) (logid:e76a82b2) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:46:28,303 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) Seq 1-4725683383995203874: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:46:28,339 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:46:28,340 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) Seq 2-3926857400090362177: Routed from 90520738888109 2025-06-29 04:46:28,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8169406e]) (logid:97225ffc) No inactive management server node found 2025-06-29 04:46:28,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8169406e]) (logid:97225ffc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:28,426 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-322bb360]) (logid:7082073a) Seq 2-3926857400090362177: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:46:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-225c49e6]) (logid:57316f40) No inactive management server node found 2025-06-29 04:46:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-225c49e6]) (logid:57316f40) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:30,024 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) StorageCollector is running... 2025-06-29 04:46:30,031 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:46:30,031 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:46:30,031 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:46:30,032 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3938116399158788173: Routed from 90520738888109 2025-06-29 04:46:30,088 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 4-3938116399158788173: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:46:30,091 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:46:30,091 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:46:30,091 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:46:30,093 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3926857400090362178: Routed from 90520738888109 2025-06-29 04:46:30,105 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3926857400090362178: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:46:30,115 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:46:30,115 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:46:30,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:46:30,116 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3926857400090362179: Routed from 90520738888109 2025-06-29 04:46:30,210 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3926857400090362179: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:46:30,220 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:46:30,220 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:46:30,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:46:30,222 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-4725683383995203875: Routed from 90520738888109 2025-06-29 04:46:30,265 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 1-4725683383995203875: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:46:30,268 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:46:30,268 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:46:30,268 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:46:30,269 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3926857400090362180: Routed from 90520738888109 2025-06-29 04:46:30,314 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-0accfaa4]) (logid:f902139a) Seq 2-3926857400090362180: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:46:30,393 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8759: Seq 7-8759: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:30,402 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8759 2025-06-29 04:46:30,798 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c78dfe8d]) (logid:babbf636) HA health check task is running... 2025-06-29 04:46:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11ed5af4]) (logid:353c49f1) No inactive management server node found 2025-06-29 04:46:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11ed5af4]) (logid:353c49f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2752cb9d]) (logid:33858d91) No inactive management server node found 2025-06-29 04:46:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2752cb9d]) (logid:33858d91) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adf81b5e]) (logid:7360d381) No inactive management server node found 2025-06-29 04:46:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adf81b5e]) (logid:7360d381) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:34,799 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f334a052]) (logid:a114c458) HA health check task is running... 2025-06-29 04:46:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ac839c9]) (logid:d63d9516) No inactive management server node found 2025-06-29 04:46:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ac839c9]) (logid:d63d9516) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55f6a33e]) (logid:18bf86f7) No inactive management server node found 2025-06-29 04:46:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55f6a33e]) (logid:18bf86f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:38,800 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b739240b]) (logid:d9ad42db) HA health check task is running... 2025-06-29 04:46:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad598481]) (logid:8072af28) No inactive management server node found 2025-06-29 04:46:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad598481]) (logid:8072af28) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5ecfb03]) (logid:5f8d435b) No inactive management server node found 2025-06-29 04:46:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f5ecfb03]) (logid:5f8d435b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:40,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8760: Seq 7-8760: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:40,403 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8760 2025-06-29 04:46:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a8570e7]) (logid:f260e800) No inactive management server node found 2025-06-29 04:46:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a8570e7]) (logid:f260e800) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:42,802 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-938b5e71]) (logid:bbf5e874) HA health check task is running... 2025-06-29 04:46:43,031 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-7c58f0f4]) (logid:8df67dde) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 04:46:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-9454bf2c]) (logid:49103e2b) DbCollector is running... 2025-06-29 04:46:43,095 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-d6821092]) (logid:c7a8cfe7) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:46:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-94bf61bf]) (logid:2e77a9b3) Found 0 running routers. 2025-06-29 04:46:43,098 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-d6821092]) (logid:c7a8cfe7) Snapshot scheduler is being called at [2025-06-29T02:46:43+0000]. 2025-06-29 04:46:43,098 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-d6821092]) (logid:c7a8cfe7) There are [0] scheduled snapshots to be executed at [2025-06-29T02:46:43+0000]. 2025-06-29 04:46:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f26bd33f]) (logid:fa3d7507) Found 0 routers to update status. 2025-06-29 04:46:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f26bd33f]) (logid:fa3d7507) Found 0 VPC's to update Redundant State. 2025-06-29 04:46:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-f26bd33f]) (logid:fa3d7507) Found 0 networks to update RvR status. 2025-06-29 04:46:43,110 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Found 1 running routers. 2025-06-29 04:46:43,115 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:46:43,115 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Seq 2-3926857400090362181: Routed from 90520738888109 2025-06-29 04:46:43,116 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Seq 2-3926857400090362181: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-ec1a2bcd]) (logid:2913b96e) External devices are not used. Skipping external device usage collection 2025-06-29 04:46:43,138 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-1283b5af]) (logid:e3f084e8) Started resource counters recalculation periodic task. 2025-06-29 04:46:43,142 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-1283b5af]) (logid:e3f084e8) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:46:43,307 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-3c3bc8af]) (logid:5efa43c3) Backup backup.poll is being called at 2025-06-29T02:46:43+0000 2025-06-29 04:46:43,313 INFO [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-cc445366]) (logid:682297fb) checking health of usage server 2025-06-29 04:46:43,315 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-1:[ctx-cc445366]) (logid:682297fb) usage server running? false, heartbeat: null 2025-06-29 04:46:43,319 WARN [c.c.a.AlertManagerImpl] (HA-1:[ctx-cc445366]) (logid:682297fb) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 04:46:43,322 DEBUG [c.c.a.AlertManagerImpl] (HA-1:[ctx-cc445366]) (logid:682297fb) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 04:46:43,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1ca8ee3]) (logid:9d62b69b) No inactive management server node found 2025-06-29 04:46:43,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c1ca8ee3]) (logid:9d62b69b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:43,556 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-1283b5af]) (logid:e3f084e8) Finished resource counters recalculation periodic task. 2025-06-29 04:46:43,656 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 2-3926857400090362181: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:43,656 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Seq 2-3926857400090362181: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:46:43,656 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 04:46:43,659 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 04:46:43,701 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 04:46:43,702 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Seq 2-3926857400090362182: Routed from 90520738888109 2025-06-29 04:46:43,703 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Seq 2-3926857400090362182: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) ManagementServerCollector is running... 2025-06-29 04:46:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) DB properties were already loaded 2025-06-29 04:46:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:46:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Execution of process [4067840] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:46:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Exit value of process [4067840] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:46:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Process [4067840] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:46:44,055 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) usage install: null 2025-06-29 04:46:44,055 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Sessions active: 0 2025-06-29 04:46:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:46:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067843] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:46:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:46:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067848] for command [/bin/bash -c uname -r ]. 2025-06-29 04:46:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:46:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067849] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:46:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) system memory from /proc: 4887515136 2025-06-29 04:46:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:46:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067854] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:46:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) free memory from /proc: 327843840 2025-06-29 04:46:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:46:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067859] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:46:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) used memory from /proc: 1019972 2025-06-29 04:46:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:46:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067860] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:46:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:46:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067861] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:46:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:46:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067862] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:46:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:46:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067869] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:46:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:46:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Successfully executed process [4067870] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail buffersdirect.count, 96 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail buffersdirect.used, 328650 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail buffersmapped.capacity, 0 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail buffersmapped.count, 0 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail buffersmapped.used, 0 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:46:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail jvmuptime, 4263829 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memoryheap.committed, 654311424 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memoryheap.init, 77594624 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memoryheap.usage, 0.2557055801521517 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorynon-heap.committed, 198770688 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorynon-heap.max, -1 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorynon-heap.usage, -1.95306504E8 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorynon-heap.used, 195306504 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11774985003998933 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21949565934615256 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012389563024044037 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.Metaspace.usage, 0.9927541855294773 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.PS-Eden-Space.usage, 0.45346100074835693 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorytotal.committed, 853082112 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorytotal.init, 85262336 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorytotal.max, 1908932607 2025-06-29 04:46:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail memorytotal.used, 683431224 2025-06-29 04:46:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail threadsdeadlocks, [] 2025-06-29 04:46:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail threadsnew.count, 0 2025-06-29 04:46:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:46:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-14ddb273]) (logid:076cdb4c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":167015376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4263759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":682602552,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":488124720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":327843840,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.14,0.12],"systemCyclesUsage":[66422279,53841649,4590245727],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:46:44,130 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 141, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":167015376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4263759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":682602552,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":488124720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":327843840,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.14,0.12],"systemCyclesUsage":[66422279,53841649,4590245727],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:46:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:46:44,135 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-30:[ctx-3ca95475]) (logid:48654998) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":167015376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4263759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":682602552,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":488124720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":327843840,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.14,0.12],"systemCyclesUsage":[66422279,53841649,4590245727],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:46:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:46:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 141, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:46:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":167015376,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4263759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":682602552,"totalCommitted":853082112,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":488124720,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":327843840,"systemMemoryUsed":1019972,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.14,0.12],"systemCyclesUsage":[66422279,53841649,4590245727],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:46:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-846dd233]) (logid:f1c0f742) No inactive management server node found 2025-06-29 04:46:44,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-846dd233]) (logid:f1c0f742) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:45,153 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 2-3926857400090362182: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.c1d431bb-dd22-4804-b797-458dbcb94ad8","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:45,153 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-49018702]) (logid:ef1b7a18) Seq 2-3926857400090362182: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 04:46:45,157 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 04:46:45,161 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:46:45,162 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Seq 2-3926857400090362183: Routed from 90520738888109 2025-06-29 04:46:45,162 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Seq 2-3926857400090362183: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:45,602 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 2-3926857400090362183: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "12.30764389038086", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "26.04389190673828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "39.63303565979004", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "53.40099334716797", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165101814", "lastRunDuration": "42.68336296081543", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165101879", "lastRunDuration": "8074.781179428101", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165110000", "lastRunDuration": "65.66500663757324", "message": "Memory Usage within limits with current at 50.851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165110066", "lastRunDuration": "3229.1502952575684", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165113349", "lastRunDuration": "54.50153350830078", "message": "Sufficient free space is 2571.23046875 MB"}, "lastRun": {"start": "2025-06-29 02:45:01.645185", "end": "2025-06-29 02:45:13.404392", "duration": "11.759206533432007"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164801246", "lastRunDuration": "41.970014572143555", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164801288", "lastRunDuration": "51.00250244140625", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164801462", "lastRunDuration": "40.079593658447266", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164801604", "lastRunDuration": "42.369842529296875", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:40:01.244231", "end": "2025-06-29 02:40:01.693885", "duration": "0.44965481758117676"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "12.30764389038086", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "26.04389190673828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "39.63303565979004", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "53.40099334716797", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165101814", "lastRunDuration": "42.68336296081543", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165101879", "lastRunDuration": "8074.781179428101", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165110000", "lastRunDuration": "65.66500663757324", "message": "Memory Usage within limits with current at 50.851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165110066", "lastRunDuration": "3229.1502952575684", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165113349", "lastRunDuration": "54.50153350830078", "message": "Sufficient free space is 2571.23046875 MB"}, "lastRun": {"start": "2025-06-29 02:45:01.645185", "end": "2025-06-29 02:45:13.404392", "duration": "11.759206533432007"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164801246", "lastRunDuration": "41.970014572143555", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164801288", "lastRunDuration": "51.00250244140625", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164801462", "lastRunDuration": "40.079593658447266", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164801604", "lastRunDuration": "42.369842529296875", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:40:01.244231", "end": "2025-06-29 02:40:01.693885", "duration": "0.44965481758117676"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:45,602 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Seq 2-3926857400090362183: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:46:45,602 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "12.30764389038086", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "26.04389190673828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "39.63303565979004", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "53.40099334716797", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165101814", "lastRunDuration": "42.68336296081543", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165101879", "lastRunDuration": "8074.781179428101", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165110000", "lastRunDuration": "65.66500663757324", "message": "Memory Usage within limits with current at 50.851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165110066", "lastRunDuration": "3229.1502952575684", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165113349", "lastRunDuration": "54.50153350830078", "message": "Sufficient free space is 2571.23046875 MB"}, "lastRun": {"start": "2025-06-29 02:45:01.645185", "end": "2025-06-29 02:45:13.404392", "duration": "11.759206533432007"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164801246", "lastRunDuration": "41.970014572143555", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164801288", "lastRunDuration": "51.00250244140625", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164801462", "lastRunDuration": "40.079593658447266", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164801604", "lastRunDuration": "42.369842529296875", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:40:01.244231", "end": "2025-06-29 02:40:01.693885", "duration": "0.44965481758117676"}}} 2025-06-29 04:46:45,633 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "12.30764389038086", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "26.04389190673828", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "39.63303565979004", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165101647", "lastRunDuration": "53.40099334716797", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165101814", "lastRunDuration": "42.68336296081543", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165101879", "lastRunDuration": "8074.781179428101", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165110000", "lastRunDuration": "65.66500663757324", "message": "Memory Usage within limits with current at 50.851%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165110066", "lastRunDuration": "3229.1502952575684", "message": "CPU Usage within limits with current at 0.6%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165113349", "lastRunDuration": "54.50153350830078", "message": "Sufficient free space is 2571.23046875 MB"}, "lastRun": {"start": "2025-06-29 02:45:01.645185", "end": "2025-06-29 02:45:13.404392", "duration": "11.759206533432007"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751164801246", "lastRunDuration": "41.970014572143555", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751164801288", "lastRunDuration": "51.00250244140625", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751164801462", "lastRunDuration": "40.079593658447266", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751164801604", "lastRunDuration": "42.369842529296875", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:40:01.244231", "end": "2025-06-29 02:40:01.693885", "duration": "0.44965481758117676"}}} 2025-06-29 04:46:45,635 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found check types executed on VR basic, start: 2025-06-29 02:45:01.645185, end: 2025-06-29 02:45:13.404392, duration: 11.759206533432007 2025-06-29 04:46:45,643 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7b015081- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 04:45:01 CEST 2025, details: service is running which took running duration (ms) 12.30764389038086 2025-06-29 04:46:45,654 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@29a140ae- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 04:45:01 CEST 2025, details: service is running which took running duration (ms) 26.04389190673828 2025-06-29 04:46:45,666 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@6708eba3- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 04:45:01 CEST 2025, details: service is running which took running duration (ms) 39.63303565979004 2025-06-29 04:46:45,675 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@292b9aef- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 04:45:01 CEST 2025, details: service is running which took running duration (ms) 53.40099334716797 2025-06-29 04:46:45,686 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@58df4213- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 04:45:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 42.68336296081543 2025-06-29 04:46:45,697 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7bd8dd96- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 04:45:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8074.781179428101 2025-06-29 04:46:45,706 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5b2d64bd- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 04:45:10 CEST 2025, details: Memory Usage within limits with current at 50.851% which took running duration (ms) 65.66500663757324 2025-06-29 04:46:45,716 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2ea471d5- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 04:45:10 CEST 2025, details: CPU Usage within limits with current at 0.6% which took running duration (ms) 3229.1502952575684 2025-06-29 04:46:45,726 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@cc961c8- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 04:45:13 CEST 2025, details: Sufficient free space is 2571.23046875 MB which took running duration (ms) 54.50153350830078 2025-06-29 04:46:45,726 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found check types executed on VR advanced, start: 2025-06-29 02:40:01.244231, end: 2025-06-29 02:40:01.693885, duration: 0.44965481758117676 2025-06-29 04:46:45,736 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@31c029aa- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 04:40:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 41.970014572143555 2025-06-29 04:46:45,747 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1f8bba3- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 04:40:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 51.00250244140625 2025-06-29 04:46:45,758 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@cdf5890- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 04:40:01 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 40.079593658447266 2025-06-29 04:46:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:46:45,768 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-921f1652]) (logid:41998a81) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4bdfafe2- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 04:40:01 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 42.369842529296875 2025-06-29 04:46:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:46:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45c79f49]) (logid:74e9fd02) No inactive management server node found 2025-06-29 04:46:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45c79f49]) (logid:74e9fd02) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:46:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:46:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:46:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:46:46,803 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-eda834f4]) (logid:c66f3f24) HA health check task is running... 2025-06-29 04:46:47,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:46:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:46:47,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:46:47,088 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:46:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa6600a6]) (logid:ab24cb17) No inactive management server node found 2025-06-29 04:46:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fa6600a6]) (logid:ab24cb17) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:48,059 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-4:[ctx-91bc3f45]) (logid:32256525) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:46:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-94a515e5]) (logid:ec8e49c2) VM scheduler.poll is being called at 2025-06-29T02:47:00+0000 2025-06-29 04:46:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-94a515e5]) (logid:ec8e49c2) Got 0 scheduled jobs to be executed at 2025-06-29T02:47:00+0000 2025-06-29 04:46:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-94a515e5]) (logid:ec8e49c2) Cleaned up 0 VM scheduled job entries 2025-06-29 04:46:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-67bb2b2a]) (logid:504edaf3) VM Operation Thread Running 2025-06-29 04:46:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f7f18ed9]) (logid:836927f3) ClusterDRS.poll is being called at 2025-06-29T02:47:00+0000 2025-06-29 04:46:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f7f18ed9]) (logid:836927f3) Removed 0 old drs migration plans 2025-06-29 04:46:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13a13a43]) (logid:badd9fc1) No inactive management server node found 2025-06-29 04:46:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13a13a43]) (logid:badd9fc1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:50,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8762: Seq 7-8762: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:46:50,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8762 2025-06-29 04:46:50,805 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c91685f9]) (logid:5d7f730a) HA health check task is running... 2025-06-29 04:46:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60a98b4a]) (logid:55fc4bbb) No inactive management server node found 2025-06-29 04:46:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60a98b4a]) (logid:55fc4bbb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5e4429f]) (logid:044dd848) No inactive management server node found 2025-06-29 04:46:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c5e4429f]) (logid:044dd848) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:53,151 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-a8dcfe29]) (logid:8ca0997c) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:46:53,162 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-a8dcfe29]) (logid:8ca0997c) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:46:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-706ec450]) (logid:6c41c401) No inactive management server node found 2025-06-29 04:46:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-706ec450]) (logid:6c41c401) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:54,807 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-0ad15b01]) (logid:765890ca) HA health check task is running... 2025-06-29 04:46:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14d50df8]) (logid:8bb057d7) No inactive management server node found 2025-06-29 04:46:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14d50df8]) (logid:8bb057d7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa264581]) (logid:b61fb2db) No inactive management server node found 2025-06-29 04:46:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa264581]) (logid:b61fb2db) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-dd798a07]) (logid:889c502d) Enabled non-edge zones available for scan: 1 2025-06-29 04:46:58,091 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-dd798a07]) (logid:889c502d) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:46:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-dd798a07]) (logid:889c502d) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:46:58,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:46:58,097 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Seq 2-4725683383995203876: Routed from 90520738888109 2025-06-29 04:46:58,140 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Seq 1-4725683383995203876: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:46:58,141 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 04:46:58,168 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:46:58,170 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Seq 2-3926857400090362184: Routed from 90520738888109 2025-06-29 04:46:58,215 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Seq 2-3926857400090362184: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:46:58,215 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-eb101ceb]) (logid:4f22a3ea) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 04:46:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-01397028]) (logid:124f29d1) Enabled non-edge zones available for scan: 1 2025-06-29 04:46:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-01397028]) (logid:124f29d1) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:46:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb77653f]) (logid:100eae31) No inactive management server node found 2025-06-29 04:46:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb77653f]) (logid:100eae31) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:46:58,807 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8e6af1ab]) (logid:e5780cbb) HA health check task is running... 2025-06-29 04:46:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-959fea6a]) (logid:40e70519) No inactive management server node found 2025-06-29 04:46:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-959fea6a]) (logid:40e70519) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:00,394 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8763: Seq 7-8763: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:00,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8763 2025-06-29 04:47:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7871b9ee]) (logid:dafc4907) No inactive management server node found 2025-06-29 04:47:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7871b9ee]) (logid:dafc4907) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:02,808 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cb1cf2b6]) (logid:fafe2b62) HA health check task is running... 2025-06-29 04:47:02,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e51274c7]) (logid:550569a8) No inactive management server node found 2025-06-29 04:47:02,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e51274c7]) (logid:550569a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5f16807]) (logid:4def8576) No inactive management server node found 2025-06-29 04:47:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5f16807]) (logid:4def8576) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67e87d09]) (logid:e53cde77) No inactive management server node found 2025-06-29 04:47:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67e87d09]) (logid:e53cde77) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:06,810 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a9980cbd]) (logid:35033328) HA health check task is running... 2025-06-29 04:47:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ff35b89]) (logid:4f05f8a2) No inactive management server node found 2025-06-29 04:47:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ff35b89]) (logid:4f05f8a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9619708a]) (logid:1d451bb3) No inactive management server node found 2025-06-29 04:47:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9619708a]) (logid:1d451bb3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca3e559d]) (logid:31b91a8b) No inactive management server node found 2025-06-29 04:47:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca3e559d]) (logid:31b91a8b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:10,395 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8764: Seq 7-8764: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:10,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8764 2025-06-29 04:47:10,811 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-26ef60ae]) (logid:874cd3bd) HA health check task is running... 2025-06-29 04:47:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cf42d46]) (logid:91170f4f) No inactive management server node found 2025-06-29 04:47:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9cf42d46]) (logid:91170f4f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-7fb2e6ce]) (logid:2ea09acd) Checking if any host reservation can be released ... 2025-06-29 04:47:13,012 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-7fb2e6ce]) (logid:2ea09acd) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-7fb2e6ce]) (logid:2ea09acd) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-7fb2e6ce]) (logid:2ea09acd) Done running HostReservationReleaseChecker ... 2025-06-29 04:47:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-08a3dadb]) (logid:4a92eb3d) Found 0 routers to update status. 2025-06-29 04:47:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-08a3dadb]) (logid:4a92eb3d) Found 0 VPC's to update Redundant State. 2025-06-29 04:47:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-08a3dadb]) (logid:4a92eb3d) Found 0 networks to update RvR status. 2025-06-29 04:47:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Running Capacity Checker ... 2025-06-29 04:47:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) recalculating system capacity 2025-06-29 04:47:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Executing cpu/ram capacity update 2025-06-29 04:47:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:13,174 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:13,185 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:47:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:47:13,191 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:47:13,203 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:13,204 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:13,211 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:47:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:47:13,212 DEBUG [c.c.c.CapacityManagerImpl] (pool-40-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:47:13,220 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Done executing cpu/ram capacity update 2025-06-29 04:47:13,220 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Executing storage capacity update 2025-06-29 04:47:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:47:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:47:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:47:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:47:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:47:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:47:13,250 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:47:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:47:13,263 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:47:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:47:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:47:13,270 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:47:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:47:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:47:13,277 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:47:13,282 DEBUG [c.c.s.StorageManagerImpl] (pool-41-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:47:13,282 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Done executing storage capacity update 2025-06-29 04:47:13,282 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Executing capacity updates for public ip and Vlans 2025-06-29 04:47:13,299 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Done capacity updates for public ip and Vlans 2025-06-29 04:47:13,299 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Executing capacity updates for private ip 2025-06-29 04:47:13,304 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Done executing capacity updates for private ip 2025-06-29 04:47:13,304 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Done recalculating system capacity 2025-06-29 04:47:13,350 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-d19d5877]) (logid:23ac30ab) Done running Capacity Checker ... 2025-06-29 04:47:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39b69443]) (logid:4099dc48) No inactive management server node found 2025-06-29 04:47:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39b69443]) (logid:4099dc48) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:14,812 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-32da1dd1]) (logid:909b097c) HA health check task is running... 2025-06-29 04:47:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfe7057d]) (logid:80b821f3) No inactive management server node found 2025-06-29 04:47:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfe7057d]) (logid:80b821f3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93558873]) (logid:a4465f8a) No inactive management server node found 2025-06-29 04:47:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93558873]) (logid:a4465f8a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61df937a]) (logid:bbc06c60) No inactive management server node found 2025-06-29 04:47:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61df937a]) (logid:bbc06c60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:18,080 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-78b62786]) (logid:f292b63e) AutoScaling Monitor is running... 2025-06-29 04:47:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a2481b26]) (logid:35214cac) Resetting hosts suitable for reconnect 2025-06-29 04:47:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a2481b26]) (logid:35214cac) Completed resetting hosts suitable for reconnect 2025-06-29 04:47:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a2481b26]) (logid:35214cac) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:47:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a2481b26]) (logid:35214cac) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:47:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a2481b26]) (logid:35214cac) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:47:18,099 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-a2481b26]) (logid:35214cac) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:47:18,157 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-edcd328e]) (logid:e3ba85eb) Starting to remove all volume_stats rows older than [Sat Jun 28 16:47:18 CEST 2025]. 2025-06-29 04:47:18,158 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-edcd328e]) (logid:e3ba85eb) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:47:18 CEST 2025]. 2025-06-29 04:47:18,814 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e0feef45]) (logid:d362ed25) HA health check task is running... 2025-06-29 04:47:19,160 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-4637cb42]) (logid:25af05e1) Starting to remove all vm_stats rows older than [Sat Jun 28 16:47:19 CEST 2025]. 2025-06-29 04:47:19,184 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-4637cb42]) (logid:25af05e1) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:47:19 CEST 2025]. 2025-06-29 04:47:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7461356]) (logid:bceb9529) No inactive management server node found 2025-06-29 04:47:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7461356]) (logid:bceb9529) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:20,399 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8765: Seq 7-8765: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:20,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8765 2025-06-29 04:47:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e330c565]) (logid:110a414d) No inactive management server node found 2025-06-29 04:47:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e330c565]) (logid:110a414d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:21,963 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:47:21,964 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:47:21,965 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) Seq 2-4725683383995203877: Routed from 90520738888109 2025-06-29 04:47:22,012 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) Seq 1-4725683383995203877: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:47:22,013 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:47:22,014 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) Seq 2-3926857400090362185: Routed from 90520738888109 2025-06-29 04:47:22,058 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-7f6e2c75]) (logid:9f684181) Seq 2-3926857400090362185: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:47:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d60af11e]) (logid:307a01e2) No inactive management server node found 2025-06-29 04:47:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d60af11e]) (logid:307a01e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:47:22,815 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7af2dbac]) (logid:d07df28f) HA health check task is running... 2025-06-29 04:47:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239e2431]) (logid:bbeb4b85) No inactive management server node found 2025-06-29 04:47:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239e2431]) (logid:bbeb4b85) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd0e0ac5]) (logid:699f9dd2) No inactive management server node found 2025-06-29 04:47:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd0e0ac5]) (logid:699f9dd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:26,816 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-3d3ea956]) (logid:b4e871ed) HA health check task is running... 2025-06-29 04:47:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c3bb72a]) (logid:80cfa478) No inactive management server node found 2025-06-29 04:47:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c3bb72a]) (logid:80cfa478) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8cc39a2]) (logid:659bcf5e) Enabled non-edge zones available for scan: 1 2025-06-29 04:47:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f8cc39a2]) (logid:659bcf5e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:47:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f8cc39a2]) (logid:659bcf5e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:47:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1338f39e]) (logid:a0af7fe1) Enabled non-edge zones available for scan: 1 2025-06-29 04:47:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1338f39e]) (logid:a0af7fe1) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:47:28,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a2dc870]) (logid:17e740de) No inactive management server node found 2025-06-29 04:47:28,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a2dc870]) (logid:17e740de) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:28,480 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:47:28,485 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:47:28,486 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) Seq 2-4725683383995203878: Routed from 90520738888109 2025-06-29 04:47:28,541 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) Seq 1-4725683383995203878: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:47:28,577 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:47:28,578 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) Seq 2-3926857400090362186: Routed from 90520738888109 2025-06-29 04:47:28,666 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-5ee2fb0c]) (logid:85c33bf8) Seq 2-3926857400090362186: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:47:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bed2af71]) (logid:009ee3a2) No inactive management server node found 2025-06-29 04:47:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bed2af71]) (logid:009ee3a2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:30,314 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) StorageCollector is running... 2025-06-29 04:47:30,326 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:47:30,326 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:47:30,326 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:47:30,327 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-3938116399158788174: Routed from 90520738888109 2025-06-29 04:47:30,380 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 4-3938116399158788174: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:47:30,384 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:47:30,384 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:47:30,384 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:47:30,386 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-3926857400090362187: Routed from 90520738888109 2025-06-29 04:47:30,396 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8766: Seq 7-8766: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:30,397 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-3926857400090362187: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:47:30,406 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8766 2025-06-29 04:47:30,408 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:47:30,408 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:47:30,408 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:47:30,408 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-4725683383995203879: Routed from 90520738888109 2025-06-29 04:47:30,456 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 1-4725683383995203879: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:47:30,469 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:47:30,469 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:47:30,469 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:47:30,470 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-4725683383995203880: Routed from 90520738888109 2025-06-29 04:47:30,555 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 1-4725683383995203880: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:47:30,557 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:47:30,557 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:47:30,557 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:47:30,558 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-3926857400090362188: Routed from 90520738888109 2025-06-29 04:47:30,603 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-d96acefd]) (logid:6622cca3) Seq 2-3926857400090362188: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:47:30,818 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5d317509]) (logid:d1c97e25) HA health check task is running... 2025-06-29 04:47:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8c734e9]) (logid:406ecdd1) No inactive management server node found 2025-06-29 04:47:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8c734e9]) (logid:406ecdd1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:32,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-143deaff]) (logid:bb82d316) No inactive management server node found 2025-06-29 04:47:32,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-143deaff]) (logid:bb82d316) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aec6567c]) (logid:78d6af88) No inactive management server node found 2025-06-29 04:47:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aec6567c]) (logid:78d6af88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:34,819 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c9a7bdf1]) (logid:2e579cd7) HA health check task is running... 2025-06-29 04:47:35,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8767: Seq 7-8767: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:35,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8767 2025-06-29 04:47:35,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0582a44f]) (logid:d20e01ce) No inactive management server node found 2025-06-29 04:47:35,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0582a44f]) (logid:d20e01ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99ea0d4a]) (logid:b6cbe65e) No inactive management server node found 2025-06-29 04:47:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99ea0d4a]) (logid:b6cbe65e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:38,820 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-f96bce6a]) (logid:90773047) HA health check task is running... 2025-06-29 04:47:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fce114e]) (logid:d24d2edb) No inactive management server node found 2025-06-29 04:47:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7fce114e]) (logid:d24d2edb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49f4be03]) (logid:cea15a7e) No inactive management server node found 2025-06-29 04:47:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49f4be03]) (logid:cea15a7e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7e8b3d8]) (logid:dbaa165f) No inactive management server node found 2025-06-29 04:47:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7e8b3d8]) (logid:dbaa165f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:42,821 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-9be1cbba]) (logid:e7927dda) HA health check task is running... 2025-06-29 04:47:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-bb22d106]) (logid:c8676889) DbCollector is running... 2025-06-29 04:47:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-595435d6]) (logid:ea4e6be6) Found 0 routers to update status. 2025-06-29 04:47:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-595435d6]) (logid:ea4e6be6) Found 0 VPC's to update Redundant State. 2025-06-29 04:47:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-595435d6]) (logid:ea4e6be6) Found 0 networks to update RvR status. 2025-06-29 04:47:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e768cc95]) (logid:52af44f1) No inactive management server node found 2025-06-29 04:47:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e768cc95]) (logid:52af44f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) ManagementServerCollector is running... 2025-06-29 04:47:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) DB properties were already loaded 2025-06-29 04:47:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:47:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Execution of process [4067884] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:47:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Exit value of process [4067884] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:47:44,058 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Process [4067884] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:47:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) usage install: null 2025-06-29 04:47:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Sessions active: 0 2025-06-29 04:47:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:47:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067887] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:47:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:47:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067892] for command [/bin/bash -c uname -r ]. 2025-06-29 04:47:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:47:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067893] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:47:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) system memory from /proc: 4887515136 2025-06-29 04:47:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:47:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067898] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:47:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) free memory from /proc: 332238848 2025-06-29 04:47:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:47:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067903] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:47:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) used memory from /proc: 1020228 2025-06-29 04:47:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:47:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067904] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:47:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:47:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067905] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:47:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:47:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067906] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:47:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:47:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067913] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:47:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:47:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Successfully executed process [4067914] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail buffersdirect.count, 96 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail buffersdirect.used, 328650 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail buffersmapped.capacity, 0 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail buffersmapped.count, 0 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail buffersmapped.used, 0 2025-06-29 04:47:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail gcPS-Scavenge.count, 109 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail gcPS-Scavenge.time, 568 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail jvmuptime, 4323833 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memoryheap.committed, 654311424 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memoryheap.init, 77594624 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memoryheap.usage, 0.27647794677935533 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorynon-heap.committed, 198901760 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorynon-heap.max, -1 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorynon-heap.usage, -1.95488648E8 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorynon-heap.used, 195488648 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11862357537989869 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21993618155763656 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.Compressed-Class-Space.usage, 0.0123911052942276 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.Metaspace.usage, 0.9928897619806317 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.PS-Eden-Space.usage, 0.51474381415331 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.PS-Old-Gen.usage, 0.1261433420142862 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorypools.PS-Survivor-Space.usage, 0.9908936818440756 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorytotal.committed, 853213184 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorytotal.init, 85262336 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorytotal.max, 1908932607 2025-06-29 04:47:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail memorytotal.used, 723486504 2025-06-29 04:47:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail threadsdeadlocks, [] 2025-06-29 04:47:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail threadsnew.count, 0 2025-06-29 04:47:44,146 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:47:44,146 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:47:44,147 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:47:44,147 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 04:47:44,148 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-c9e180a5]) (logid:6bc0c52b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":127226704,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.77,"totalInit":85262336,"totalUsed":722572600,"totalCommitted":853213184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":527997856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":332238848,"systemMemoryUsed":1020228,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.77,0.28,0.17],"systemCyclesUsage":[66422715,53842000,4590268686],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:47:44,148 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 143, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":127226704,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.77,"totalInit":85262336,"totalUsed":722572600,"totalCommitted":853213184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":527997856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":332238848,"systemMemoryUsed":1020228,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.77,0.28,0.17],"systemCyclesUsage":[66422715,53842000,4590268686],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:47:44,148 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:47:44,154 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-31:[ctx-96b830f8]) (logid:aef86f67) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":127226704,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.77,"totalInit":85262336,"totalUsed":722572600,"totalCommitted":853213184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":527997856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":332238848,"systemMemoryUsed":1020228,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.77,0.28,0.17],"systemCyclesUsage":[66422715,53842000,4590268686],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:47:44,154 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:47:44,154 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 143, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:47:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":654311424,"freeJvmMemoryBytes":127226704,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4323762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.77,"totalInit":85262336,"totalUsed":722572600,"totalCommitted":853213184,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":527997856,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":109,"systemMemoryTotal":4887515136,"systemMemoryFree":332238848,"systemMemoryUsed":1020228,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.77,0.28,0.17],"systemCyclesUsage":[66422715,53842000,4590268686],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:47:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-847b8f4a]) (logid:61bf0b9e) No inactive management server node found 2025-06-29 04:47:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-847b8f4a]) (logid:61bf0b9e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:45,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8768: Seq 7-8768: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:45,408 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8768 2025-06-29 04:47:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:47:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:47:46,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cefc6b4]) (logid:672bf160) No inactive management server node found 2025-06-29 04:47:46,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8cefc6b4]) (logid:672bf160) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:47:46,651 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:47:46,823 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6b93c3f0]) (logid:fceb6108) HA health check task is running... 2025-06-29 04:47:47,079 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:47,082 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:47:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:47:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b431991]) (logid:8530826c) No inactive management server node found 2025-06-29 04:47:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b431991]) (logid:8530826c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-66e5f559]) (logid:0615c395) VM scheduler.poll is being called at 2025-06-29T02:48:00+0000 2025-06-29 04:47:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-66e5f559]) (logid:0615c395) Got 0 scheduled jobs to be executed at 2025-06-29T02:48:00+0000 2025-06-29 04:47:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-66e5f559]) (logid:0615c395) Cleaned up 0 VM scheduled job entries 2025-06-29 04:47:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-713d4b56]) (logid:7af5fa7f) VM Operation Thread Running 2025-06-29 04:47:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-93990c60]) (logid:c784098b) ClusterDRS.poll is being called at 2025-06-29T02:48:00+0000 2025-06-29 04:47:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-93990c60]) (logid:c784098b) Removed 0 old drs migration plans 2025-06-29 04:47:49,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6db0b15d]) (logid:6e08da71) No inactive management server node found 2025-06-29 04:47:49,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6db0b15d]) (logid:6e08da71) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:50,824 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-84fe6c40]) (logid:c7f1e690) HA health check task is running... 2025-06-29 04:47:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3dd9f55]) (logid:4946e516) No inactive management server node found 2025-06-29 04:47:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a3dd9f55]) (logid:4946e516) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01d8fe29]) (logid:4fa66c43) No inactive management server node found 2025-06-29 04:47:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01d8fe29]) (logid:4fa66c43) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a7ba229]) (logid:1977cc3b) No inactive management server node found 2025-06-29 04:47:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a7ba229]) (logid:1977cc3b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:54,825 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-398d23fd]) (logid:b0101b5b) HA health check task is running... 2025-06-29 04:47:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a64748d]) (logid:4e59c54a) No inactive management server node found 2025-06-29 04:47:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a64748d]) (logid:4e59c54a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:55,397 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8770: Seq 7-8770: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:47:55,406 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8770 2025-06-29 04:47:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-580c251f]) (logid:55f1daa4) No inactive management server node found 2025-06-29 04:47:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-580c251f]) (logid:55f1daa4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67805c9e]) (logid:b34a1197) Enabled non-edge zones available for scan: 1 2025-06-29 04:47:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-67805c9e]) (logid:b34a1197) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:47:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-67805c9e]) (logid:b34a1197) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:47:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2ef52a34]) (logid:b874d207) Enabled non-edge zones available for scan: 1 2025-06-29 04:47:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-2ef52a34]) (logid:b874d207) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:47:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94c50a98]) (logid:f93952e8) No inactive management server node found 2025-06-29 04:47:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94c50a98]) (logid:f93952e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:47:58,827 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3600e946]) (logid:bc96ea7f) HA health check task is running... 2025-06-29 04:47:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c830242]) (logid:77efd427) No inactive management server node found 2025-06-29 04:47:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c830242]) (logid:77efd427) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67cf2a87]) (logid:abfb0648) No inactive management server node found 2025-06-29 04:48:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67cf2a87]) (logid:abfb0648) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:02,828 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-749c7cbd]) (logid:6a63c74e) HA health check task is running... 2025-06-29 04:48:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30b287a3]) (logid:1d16babc) No inactive management server node found 2025-06-29 04:48:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30b287a3]) (logid:1d16babc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3749f6bf]) (logid:52590284) No inactive management server node found 2025-06-29 04:48:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3749f6bf]) (logid:52590284) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:05,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8771: Seq 7-8771: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:48:05,410 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8771 2025-06-29 04:48:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b79d1e72]) (logid:f571fae1) No inactive management server node found 2025-06-29 04:48:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b79d1e72]) (logid:f571fae1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:06,829 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8ea63b17]) (logid:d9f975e1) HA health check task is running... 2025-06-29 04:48:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd2fd500]) (logid:e3c8b7b6) No inactive management server node found 2025-06-29 04:48:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd2fd500]) (logid:e3c8b7b6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47a12add]) (logid:e307424e) No inactive management server node found 2025-06-29 04:48:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-47a12add]) (logid:e307424e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b646b872]) (logid:2dff26ee) No inactive management server node found 2025-06-29 04:48:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b646b872]) (logid:2dff26ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:10,831 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b6f5dc77]) (logid:261255e4) HA health check task is running... 2025-06-29 04:48:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95061806]) (logid:b33ba5c9) No inactive management server node found 2025-06-29 04:48:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95061806]) (logid:b33ba5c9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:48:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85590810]) (logid:2aa14ddf) Found 0 routers to update status. 2025-06-29 04:48:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85590810]) (logid:2aa14ddf) Found 0 VPC's to update Redundant State. 2025-06-29 04:48:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-85590810]) (logid:2aa14ddf) Found 0 networks to update RvR status. 2025-06-29 04:48:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-699c3910]) (logid:fc3f1b78) No inactive management server node found 2025-06-29 04:48:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-699c3910]) (logid:fc3f1b78) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:14,832 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-6c2cd8f0]) (logid:4e9b4f8f) HA health check task is running... 2025-06-29 04:48:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de8d8209]) (logid:697fca5e) No inactive management server node found 2025-06-29 04:48:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-de8d8209]) (logid:697fca5e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:15,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8772: Seq 7-8772: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:48:15,407 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8772 2025-06-29 04:48:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b727772]) (logid:03e206ab) No inactive management server node found 2025-06-29 04:48:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b727772]) (logid:03e206ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57f7fd01]) (logid:dae91a9e) No inactive management server node found 2025-06-29 04:48:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57f7fd01]) (logid:dae91a9e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:18,082 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-4bd9fa63]) (logid:e5edc10b) AutoScaling Monitor is running... 2025-06-29 04:48:18,161 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2f11f110]) (logid:f035177c) Starting to remove all volume_stats rows older than [Sat Jun 28 16:48:18 CEST 2025]. 2025-06-29 04:48:18,161 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-2f11f110]) (logid:f035177c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:48:18 CEST 2025]. 2025-06-29 04:48:18,833 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-954dd385]) (logid:43d44654) HA health check task is running... 2025-06-29 04:48:19,185 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-6624bc65]) (logid:9efd7e47) Starting to remove all vm_stats rows older than [Sat Jun 28 16:48:19 CEST 2025]. 2025-06-29 04:48:19,207 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-6624bc65]) (logid:9efd7e47) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:48:19 CEST 2025]. 2025-06-29 04:48:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba83a314]) (logid:897c696a) No inactive management server node found 2025-06-29 04:48:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba83a314]) (logid:897c696a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a28b397e]) (logid:1b0c31e9) No inactive management server node found 2025-06-29 04:48:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a28b397e]) (logid:1b0c31e9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:22,061 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:48:22,063 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:48:22,064 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) Seq 2-4725683383995203881: Routed from 90520738888109 2025-06-29 04:48:22,108 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) Seq 1-4725683383995203881: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:48:22,109 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:48:22,110 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) Seq 2-3926857400090362189: Routed from 90520738888109 2025-06-29 04:48:22,154 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bac49eaa]) (logid:3d1ba3df) Seq 2-3926857400090362189: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:48:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d58e5ebd]) (logid:a6bb2901) No inactive management server node found 2025-06-29 04:48:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d58e5ebd]) (logid:a6bb2901) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:22,834 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-79f3eec8]) (logid:3f00cbb8) HA health check task is running... 2025-06-29 04:48:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e73a495]) (logid:35a9c9c5) No inactive management server node found 2025-06-29 04:48:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e73a495]) (logid:35a9c9c5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-122b08fd]) (logid:e4b5d44a) No inactive management server node found 2025-06-29 04:48:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-122b08fd]) (logid:e4b5d44a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:25,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8773: Seq 7-8773: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:48:25,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8773 2025-06-29 04:48:26,836 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-51b256b9]) (logid:10a4b30b) HA health check task is running... 2025-06-29 04:48:26,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13627290]) (logid:f9aab2fb) No inactive management server node found 2025-06-29 04:48:26,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13627290]) (logid:f9aab2fb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d06a40a0]) (logid:59037bfc) Enabled non-edge zones available for scan: 1 2025-06-29 04:48:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d06a40a0]) (logid:59037bfc) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:48:28,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d06a40a0]) (logid:59037bfc) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:48:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ece6f6ff]) (logid:727f7cb8) Enabled non-edge zones available for scan: 1 2025-06-29 04:48:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-ece6f6ff]) (logid:727f7cb8) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:48:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6843687a]) (logid:319c8ccd) No inactive management server node found 2025-06-29 04:48:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6843687a]) (logid:319c8ccd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:28,723 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:48:28,726 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:48:28,727 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) Seq 2-4725683383995203882: Routed from 90520738888109 2025-06-29 04:48:28,779 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) Seq 1-4725683383995203882: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:48:28,814 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:48:28,814 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) Seq 2-3926857400090362190: Routed from 90520738888109 2025-06-29 04:48:28,901 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-a7200f39]) (logid:065a16b0) Seq 2-3926857400090362190: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:48:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a69ee367]) (logid:ba113b21) No inactive management server node found 2025-06-29 04:48:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a69ee367]) (logid:ba113b21) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:30,604 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) StorageCollector is running... 2025-06-29 04:48:30,609 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:48:30,610 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:48:30,610 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:48:30,611 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-3938116399158788175: Routed from 90520738888109 2025-06-29 04:48:30,666 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 4-3938116399158788175: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:48:30,670 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:48:30,670 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:48:30,671 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:48:30,672 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-4725683383995203883: Routed from 90520738888109 2025-06-29 04:48:30,719 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 1-4725683383995203883: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:48:30,729 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:48:30,729 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:48:30,729 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:48:30,731 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-3926857400090362191: Routed from 90520738888109 2025-06-29 04:48:30,742 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-3926857400090362191: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:48:30,752 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:48:30,752 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:48:30,752 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:48:30,753 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-4725683383995203884: Routed from 90520738888109 2025-06-29 04:48:30,837 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-f0c3ada0]) (logid:c64dc612) HA health check task is running... 2025-06-29 04:48:30,838 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 1-4725683383995203884: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:48:30,839 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:48:30,839 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:48:30,839 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:48:30,840 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-3926857400090362192: Routed from 90520738888109 2025-06-29 04:48:30,884 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-687c2d81]) (logid:a15fc86b) Seq 2-3926857400090362192: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:48:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee4cb182]) (logid:f327b4a4) No inactive management server node found 2025-06-29 04:48:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee4cb182]) (logid:f327b4a4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbf5b822]) (logid:176609a9) No inactive management server node found 2025-06-29 04:48:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbf5b822]) (logid:176609a9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85788084]) (logid:7f10995f) No inactive management server node found 2025-06-29 04:48:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-85788084]) (logid:7f10995f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:34,838 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3a3f7451]) (logid:2d109b2f) HA health check task is running... 2025-06-29 04:48:35,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8774: Seq 7-8774: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:48:35,410 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8774 2025-06-29 04:48:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04665207]) (logid:3b6a52d0) No inactive management server node found 2025-06-29 04:48:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04665207]) (logid:3b6a52d0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c89169a1]) (logid:1c84536b) No inactive management server node found 2025-06-29 04:48:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c89169a1]) (logid:1c84536b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:38,839 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-3c6667dd]) (logid:b3517cb0) HA health check task is running... 2025-06-29 04:48:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd5dbb9f]) (logid:2570fd71) No inactive management server node found 2025-06-29 04:48:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cd5dbb9f]) (logid:2570fd71) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa33ef21]) (logid:51e12cf4) No inactive management server node found 2025-06-29 04:48:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa33ef21]) (logid:51e12cf4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-957f7861]) (logid:74215c5f) No inactive management server node found 2025-06-29 04:48:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-957f7861]) (logid:74215c5f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:42,841 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-a269e4df]) (logid:e0d1221b) HA health check task is running... 2025-06-29 04:48:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-0557052e]) (logid:792810b8) DbCollector is running... 2025-06-29 04:48:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e588a256]) (logid:3b3803e2) Found 0 routers to update status. 2025-06-29 04:48:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e588a256]) (logid:3b3803e2) Found 0 VPC's to update Redundant State. 2025-06-29 04:48:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-e588a256]) (logid:3b3803e2) Found 0 networks to update RvR status. 2025-06-29 04:48:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8444e55]) (logid:07677a7b) No inactive management server node found 2025-06-29 04:48:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c8444e55]) (logid:07677a7b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) ManagementServerCollector is running... 2025-06-29 04:48:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) DB properties were already loaded 2025-06-29 04:48:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:48:44,055 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Execution of process [4067924] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:48:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Exit value of process [4067924] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:48:44,055 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Process [4067924] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:48:44,055 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) usage install: null 2025-06-29 04:48:44,055 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Sessions active: 0 2025-06-29 04:48:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:48:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067927] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:48:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:48:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067932] for command [/bin/bash -c uname -r ]. 2025-06-29 04:48:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:48:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067933] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:48:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) system memory from /proc: 4887515136 2025-06-29 04:48:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:48:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067938] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:48:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) free memory from /proc: 351444992 2025-06-29 04:48:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:48:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067943] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:48:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) used memory from /proc: 989936 2025-06-29 04:48:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:48:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067944] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:48:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:48:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067945] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:48:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:48:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067946] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:48:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:48:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067953] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:48:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:48:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Successfully executed process [4067954] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail buffersdirect.count, 96 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail buffersdirect.used, 328650 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail buffersmapped.capacity, 0 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail buffersmapped.count, 0 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail buffersmapped.used, 0 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail jvmuptime, 4383828 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memoryheap.committed, 671088640 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memoryheap.init, 77594624 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memoryheap.usage, 0.12065502943098136 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorynon-heap.committed, 198901760 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorynon-heap.max, -1 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorynon-heap.usage, -1.95495512E8 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorynon-heap.used, 195495512 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11862982371367635 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.21997263138600992 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.Compressed-Class-Space.usage, 0.0123911052942276 2025-06-29 04:48:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.Metaspace.usage, 0.9929013223196496 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.PS-Eden-Space.usage, 0.04440401975850539 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorytotal.committed, 869990400 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorytotal.init, 85262336 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorytotal.max, 1908932607 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail memorytotal.used, 426000328 2025-06-29 04:48:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail threadsdeadlocks, [] 2025-06-29 04:48:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail threadsnew.count, 0 2025-06-29 04:48:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:48:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-db1a224a]) (logid:04796c68) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":443333488,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4383758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.72,"totalInit":85262336,"totalUsed":423250664,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":230322320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351444992,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.72,0.37,0.21],"systemCyclesUsage":[66423031,53842296,4590291829],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:48:44,129 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 145, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":443333488,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4383758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.72,"totalInit":85262336,"totalUsed":423250664,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":230322320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351444992,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.72,0.37,0.21],"systemCyclesUsage":[66423031,53842296,4590291829],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:48:44,129 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:48:44,134 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-31:[ctx-730b1d93]) (logid:354f9598) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":443333488,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4383758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.72,"totalInit":85262336,"totalUsed":423250664,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":230322320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351444992,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.72,0.37,0.21],"systemCyclesUsage":[66423031,53842296,4590291829],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:48:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:48:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 145, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:48:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":443333488,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4383758,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.72,"totalInit":85262336,"totalUsed":423250664,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":230322320,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351444992,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.72,0.37,0.21],"systemCyclesUsage":[66423031,53842296,4590291829],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:48:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52869bb9]) (logid:b0b9244f) No inactive management server node found 2025-06-29 04:48:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-52869bb9]) (logid:b0b9244f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:45,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8775: Seq 7-8775: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:48:45,410 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8775 2025-06-29 04:48:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:48:45,925 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:48:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce1168ec]) (logid:00eeb8ab) No inactive management server node found 2025-06-29 04:48:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce1168ec]) (logid:00eeb8ab) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:48:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:48:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:48:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:48:46,843 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-2678aaec]) (logid:ef24dad3) HA health check task is running... 2025-06-29 04:48:47,087 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:48:47,088 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:48:47,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:48:47,092 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:48:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80ffbf35]) (logid:9dc74627) No inactive management server node found 2025-06-29 04:48:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80ffbf35]) (logid:9dc74627) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c3c78263]) (logid:9c238cbc) Resetting hosts suitable for reconnect 2025-06-29 04:48:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c3c78263]) (logid:9c238cbc) Completed resetting hosts suitable for reconnect 2025-06-29 04:48:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c3c78263]) (logid:9c238cbc) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:48:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c3c78263]) (logid:9c238cbc) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:48:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c3c78263]) (logid:9c238cbc) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:48:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-c3c78263]) (logid:9c238cbc) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:48:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-81bfb02e]) (logid:7e2cefe9) VM scheduler.poll is being called at 2025-06-29T02:49:00+0000 2025-06-29 04:48:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-81bfb02e]) (logid:7e2cefe9) Got 0 scheduled jobs to be executed at 2025-06-29T02:49:00+0000 2025-06-29 04:48:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-81bfb02e]) (logid:7e2cefe9) Cleaned up 0 VM scheduled job entries 2025-06-29 04:48:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-bf8a9bc0]) (logid:67e3eca6) VM Operation Thread Running 2025-06-29 04:48:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-b04faab9]) (logid:369e5455) ClusterDRS.poll is being called at 2025-06-29T02:49:00+0000 2025-06-29 04:48:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-b04faab9]) (logid:369e5455) Removed 0 old drs migration plans 2025-06-29 04:48:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cafec3bc]) (logid:936659d3) No inactive management server node found 2025-06-29 04:48:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cafec3bc]) (logid:936659d3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:50,844 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-a563202e]) (logid:e7a10fae) HA health check task is running... 2025-06-29 04:48:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321b2524]) (logid:57542ce7) No inactive management server node found 2025-06-29 04:48:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321b2524]) (logid:57542ce7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d1a97a2]) (logid:21d7ee5b) No inactive management server node found 2025-06-29 04:48:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d1a97a2]) (logid:21d7ee5b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9585c2ca]) (logid:109e2789) No inactive management server node found 2025-06-29 04:48:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9585c2ca]) (logid:109e2789) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:54,845 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1e86b3fa]) (logid:de729deb) HA health check task is running... 2025-06-29 04:48:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ef19545]) (logid:b9acf003) No inactive management server node found 2025-06-29 04:48:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8ef19545]) (logid:b9acf003) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:55,400 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8777: Seq 7-8777: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:48:55,410 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8777 2025-06-29 04:48:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7d649ce]) (logid:a4a0b3ad) No inactive management server node found 2025-06-29 04:48:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7d649ce]) (logid:a4a0b3ad) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4f724f29]) (logid:fe8ad69c) Enabled non-edge zones available for scan: 1 2025-06-29 04:48:58,091 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4f724f29]) (logid:fe8ad69c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:48:58,092 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4f724f29]) (logid:fe8ad69c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:48:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-523759f3]) (logid:6fb9793e) Enabled non-edge zones available for scan: 1 2025-06-29 04:48:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-523759f3]) (logid:6fb9793e) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:48:58,357 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e3a27b3]) (logid:d35ef687) No inactive management server node found 2025-06-29 04:48:58,357 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e3a27b3]) (logid:d35ef687) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:48:58,846 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-8ff79b6c]) (logid:5b63cbc7) HA health check task is running... 2025-06-29 04:48:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3234bb7d]) (logid:470d14e4) No inactive management server node found 2025-06-29 04:48:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3234bb7d]) (logid:470d14e4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-605a8514]) (logid:af9c38c7) No inactive management server node found 2025-06-29 04:49:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-605a8514]) (logid:af9c38c7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:02,848 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-51cfe60f]) (logid:c960d397) HA health check task is running... 2025-06-29 04:49:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8beae024]) (logid:f4114ccb) No inactive management server node found 2025-06-29 04:49:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8beae024]) (logid:f4114ccb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c3ec51b]) (logid:a6a2a465) No inactive management server node found 2025-06-29 04:49:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c3ec51b]) (logid:a6a2a465) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:05,402 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8778: Seq 7-8778: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:49:05,411 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8778 2025-06-29 04:49:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b44375ef]) (logid:4afd2a6f) No inactive management server node found 2025-06-29 04:49:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b44375ef]) (logid:4afd2a6f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:06,849 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d3c39b6c]) (logid:49ac10c8) HA health check task is running... 2025-06-29 04:49:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c79042b]) (logid:0d538a69) No inactive management server node found 2025-06-29 04:49:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c79042b]) (logid:0d538a69) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92ca682d]) (logid:8eec9c3a) No inactive management server node found 2025-06-29 04:49:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-92ca682d]) (logid:8eec9c3a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0f44a32]) (logid:dcb26182) No inactive management server node found 2025-06-29 04:49:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0f44a32]) (logid:dcb26182) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:10,850 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2ab70b6f]) (logid:cf58e2c3) HA health check task is running... 2025-06-29 04:49:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae1cfc70]) (logid:2e6422df) No inactive management server node found 2025-06-29 04:49:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae1cfc70]) (logid:2e6422df) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-03483c63]) (logid:afeddd20) Found 0 routers to update status. 2025-06-29 04:49:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-03483c63]) (logid:afeddd20) Found 0 VPC's to update Redundant State. 2025-06-29 04:49:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-03483c63]) (logid:afeddd20) Found 0 networks to update RvR status. 2025-06-29 04:49:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50c43d7c]) (logid:af366fe9) No inactive management server node found 2025-06-29 04:49:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50c43d7c]) (logid:af366fe9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:49:14,851 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-57dca3df]) (logid:febe8c41) HA health check task is running... 2025-06-29 04:49:14,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95460e97]) (logid:0c47228b) No inactive management server node found 2025-06-29 04:49:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-95460e97]) (logid:0c47228b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:15,401 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8779: Seq 7-8779: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:49:15,411 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8779 2025-06-29 04:49:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f7533a7]) (logid:d70ab8f7) No inactive management server node found 2025-06-29 04:49:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f7533a7]) (logid:d70ab8f7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b71e7516]) (logid:735f1f8e) No inactive management server node found 2025-06-29 04:49:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b71e7516]) (logid:735f1f8e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:18,085 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-3b3d08e6]) (logid:6e87f781) AutoScaling Monitor is running... 2025-06-29 04:49:18,163 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b89750e0]) (logid:2863dfc1) Starting to remove all volume_stats rows older than [Sat Jun 28 16:49:18 CEST 2025]. 2025-06-29 04:49:18,164 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b89750e0]) (logid:2863dfc1) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:49:18 CEST 2025]. 2025-06-29 04:49:18,852 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-09974234]) (logid:f45e6877) HA health check task is running... 2025-06-29 04:49:19,208 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-0143eae0]) (logid:9d16a7f6) Starting to remove all vm_stats rows older than [Sat Jun 28 16:49:19 CEST 2025]. 2025-06-29 04:49:19,267 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-0143eae0]) (logid:9d16a7f6) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:49:19 CEST 2025]. 2025-06-29 04:49:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a52e2cdc]) (logid:5310dd8f) No inactive management server node found 2025-06-29 04:49:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a52e2cdc]) (logid:5310dd8f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-477cc078]) (logid:b046f181) No inactive management server node found 2025-06-29 04:49:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-477cc078]) (logid:b046f181) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:22,157 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:49:22,159 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:49:22,160 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) Seq 2-4725683383995203885: Routed from 90520738888109 2025-06-29 04:49:22,204 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) Seq 1-4725683383995203885: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:49:22,205 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:49:22,206 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) Seq 2-3926857400090362193: Routed from 90520738888109 2025-06-29 04:49:22,250 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-3c7cda29]) (logid:8139e2b4) Seq 2-3926857400090362193: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:49:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96ce4262]) (logid:f576b0bd) No inactive management server node found 2025-06-29 04:49:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-96ce4262]) (logid:f576b0bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:22,853 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a1f01663]) (logid:9167fb14) HA health check task is running... 2025-06-29 04:49:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd165060]) (logid:e6dbe571) No inactive management server node found 2025-06-29 04:49:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dd165060]) (logid:e6dbe571) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce6abb42]) (logid:540ec4d2) No inactive management server node found 2025-06-29 04:49:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ce6abb42]) (logid:540ec4d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:25,402 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8780: Seq 7-8780: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:49:25,412 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8780 2025-06-29 04:49:26,854 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-4665ab1a]) (logid:3b4c692c) HA health check task is running... 2025-06-29 04:49:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7243bb88]) (logid:f3ac41ee) No inactive management server node found 2025-06-29 04:49:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7243bb88]) (logid:f3ac41ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fc1e1321]) (logid:3493eabf) Enabled non-edge zones available for scan: 1 2025-06-29 04:49:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-fc1e1321]) (logid:3493eabf) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:49:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-fc1e1321]) (logid:3493eabf) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:49:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3113839b]) (logid:5918fe2c) Enabled non-edge zones available for scan: 1 2025-06-29 04:49:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-3113839b]) (logid:5918fe2c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:49:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d97571b]) (logid:00ba1f2d) No inactive management server node found 2025-06-29 04:49:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1d97571b]) (logid:00ba1f2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:28,970 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:49:28,974 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:49:28,975 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) Seq 2-4725683383995203886: Routed from 90520738888109 2025-06-29 04:49:29,032 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) Seq 1-4725683383995203886: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:49:29,066 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:49:29,067 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) Seq 2-3926857400090362194: Routed from 90520738888109 2025-06-29 04:49:29,196 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-a1e16dcd]) (logid:74776871) Seq 2-3926857400090362194: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:49:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b20db5f1]) (logid:d90a9187) No inactive management server node found 2025-06-29 04:49:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b20db5f1]) (logid:d90a9187) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:30,855 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2ce85900]) (logid:cbb23b46) HA health check task is running... 2025-06-29 04:49:30,884 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) StorageCollector is running... 2025-06-29 04:49:30,890 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:49:30,890 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:49:30,890 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:49:30,891 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 2-3938116399158788176: Routed from 90520738888109 2025-06-29 04:49:30,945 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 4-3938116399158788176: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:49:30,948 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:49:30,948 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:49:30,948 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:49:30,949 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 2-4725683383995203887: Routed from 90520738888109 2025-06-29 04:49:30,997 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 1-4725683383995203887: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:49:31,010 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:49:31,010 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:49:31,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:49:31,011 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 2-4725683383995203888: Routed from 90520738888109 2025-06-29 04:49:31,100 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 1-4725683383995203888: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:49:31,110 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:49:31,111 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:49:31,111 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:49:31,111 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 2-4725683383995203889: Routed from 90520738888109 2025-06-29 04:49:31,196 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 1-4725683383995203889: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:49:31,198 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:49:31,198 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:49:31,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:49:31,199 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 2-3926857400090362195: Routed from 90520738888109 2025-06-29 04:49:31,202 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-54565d0f]) (logid:77462764) Seq 2-3926857400090362195: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:49:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0be00e03]) (logid:5095d037) No inactive management server node found 2025-06-29 04:49:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0be00e03]) (logid:5095d037) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebba680d]) (logid:a54e1d80) No inactive management server node found 2025-06-29 04:49:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebba680d]) (logid:a54e1d80) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f316e6a4]) (logid:656214ce) No inactive management server node found 2025-06-29 04:49:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f316e6a4]) (logid:656214ce) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:34,857 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5da99c3f]) (logid:a5110d24) HA health check task is running... 2025-06-29 04:49:35,406 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8781: Seq 7-8781: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:49:35,416 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8781 2025-06-29 04:49:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba62fdd9]) (logid:e3eaf9e7) No inactive management server node found 2025-06-29 04:49:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba62fdd9]) (logid:e3eaf9e7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eff635f4]) (logid:2d992f7e) No inactive management server node found 2025-06-29 04:49:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eff635f4]) (logid:2d992f7e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:38,858 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0b8dd28f]) (logid:b2fec9c0) HA health check task is running... 2025-06-29 04:49:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5796efa6]) (logid:5cabd913) No inactive management server node found 2025-06-29 04:49:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5796efa6]) (logid:5cabd913) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:49:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b661c9c]) (logid:80570440) No inactive management server node found 2025-06-29 04:49:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5b661c9c]) (logid:80570440) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5600341f]) (logid:b339700d) No inactive management server node found 2025-06-29 04:49:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5600341f]) (logid:b339700d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:42,859 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-3f66d627]) (logid:7b0d8816) HA health check task is running... 2025-06-29 04:49:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-7a00f8bd]) (logid:c6287c86) DbCollector is running... 2025-06-29 04:49:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8291a47]) (logid:7f9adb0a) Found 0 routers to update status. 2025-06-29 04:49:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8291a47]) (logid:7f9adb0a) Found 0 VPC's to update Redundant State. 2025-06-29 04:49:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-a8291a47]) (logid:7f9adb0a) Found 0 networks to update RvR status. 2025-06-29 04:49:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a731dee]) (logid:b1c6b8db) No inactive management server node found 2025-06-29 04:49:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a731dee]) (logid:b1c6b8db) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:49:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) ManagementServerCollector is running... 2025-06-29 04:49:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) DB properties were already loaded 2025-06-29 04:49:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:49:44,055 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Execution of process [4067965] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:49:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Exit value of process [4067965] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:49:44,055 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Process [4067965] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:49:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) usage install: null 2025-06-29 04:49:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Sessions active: 0 2025-06-29 04:49:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:49:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067968] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:49:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:49:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067973] for command [/bin/bash -c uname -r ]. 2025-06-29 04:49:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:49:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067974] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:49:44,072 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) system memory from /proc: 4887515136 2025-06-29 04:49:44,072 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:49:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067979] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:49:44,078 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) free memory from /proc: 357679104 2025-06-29 04:49:44,078 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:49:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067984] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:49:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) used memory from /proc: 989936 2025-06-29 04:49:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:49:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067985] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:49:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:49:44,106 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067986] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:49:44,106 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:49:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067987] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:49:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:49:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067994] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:49:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:49:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Successfully executed process [4067995] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail buffersdirect.count, 96 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail buffersdirect.used, 328650 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail buffersmapped.capacity, 0 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail buffersmapped.count, 0 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail buffersmapped.used, 0 2025-06-29 04:49:44,124 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail jvmuptime, 4443828 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memoryheap.committed, 671088640 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memoryheap.init, 77594624 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memoryheap.usage, 0.1321469175720634 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorynon-heap.committed, 198901760 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorynon-heap.max, -1 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorynon-heap.usage, -1.95535512E8 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorynon-heap.used, 195535512 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11872563149826713 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22015904622254806 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.Compressed-Class-Space.usage, 0.0123911052942276 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.Metaspace.usage, 0.9929393226419655 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.PS-Eden-Space.usage, 0.07844955655401369 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorytotal.committed, 869990400 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorytotal.init, 85262336 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorytotal.max, 1908932607 2025-06-29 04:49:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail memorytotal.used, 447795072 2025-06-29 04:49:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail threadsdeadlocks, [] 2025-06-29 04:49:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail threadsnew.count, 0 2025-06-29 04:49:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:49:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-4f45a19f]) (logid:4e9955e8) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":419662400,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":446951120,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":252259560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":357679104,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,0.39,0.22],"systemCyclesUsage":[66423358,53842609,4590314940],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:49:44,129 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 147, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":419662400,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":446951120,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":252259560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":357679104,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,0.39,0.22],"systemCyclesUsage":[66423358,53842609,4590314940],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:49:44,129 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:49:44,135 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-31:[ctx-987efc0f]) (logid:5508ef6e) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":419662400,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":446951120,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":252259560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":357679104,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,0.39,0.22],"systemCyclesUsage":[66423358,53842609,4590314940],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:49:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:49:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 147, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:49:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":419662400,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4443759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.62,"totalInit":85262336,"totalUsed":446951120,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":252259560,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":357679104,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.62,0.39,0.22],"systemCyclesUsage":[66423358,53842609,4590314940],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:49:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aec6897e]) (logid:221b16bd) No inactive management server node found 2025-06-29 04:49:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aec6897e]) (logid:221b16bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:45,403 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8782: Seq 7-8782: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:49:45,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8782 2025-06-29 04:49:45,765 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:49:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:49:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94b96d0c]) (logid:46140b37) No inactive management server node found 2025-06-29 04:49:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94b96d0c]) (logid:46140b37) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:46,651 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:49:46,653 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:49:46,654 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:49:46,656 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:49:46,861 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-cbe4588a]) (logid:4adbdc92) HA health check task is running... 2025-06-29 04:49:47,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:49:47,106 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:49:47,108 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:49:47,111 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:49:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e669492c]) (logid:3d5b9fd5) No inactive management server node found 2025-06-29 04:49:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e669492c]) (logid:3d5b9fd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-43656b15]) (logid:b9625d79) VM scheduler.poll is being called at 2025-06-29T02:50:00+0000 2025-06-29 04:49:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-43656b15]) (logid:b9625d79) Got 0 scheduled jobs to be executed at 2025-06-29T02:50:00+0000 2025-06-29 04:49:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-43656b15]) (logid:b9625d79) Cleaned up 0 VM scheduled job entries 2025-06-29 04:49:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-07e939f4]) (logid:5ffacc44) VM Operation Thread Running 2025-06-29 04:49:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4a3b39fe]) (logid:790a213a) ClusterDRS.poll is being called at 2025-06-29T02:50:00+0000 2025-06-29 04:49:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-4a3b39fe]) (logid:790a213a) Removed 0 old drs migration plans 2025-06-29 04:49:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6222d9f]) (logid:6945d920) No inactive management server node found 2025-06-29 04:49:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6222d9f]) (logid:6945d920) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69e11faf]) (logid:0c5ddafc) No inactive management server node found 2025-06-29 04:49:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-69e11faf]) (logid:0c5ddafc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:50,862 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-bf875bb1]) (logid:ea7c03c4) HA health check task is running... 2025-06-29 04:49:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75ed0959]) (logid:3a79cdd2) No inactive management server node found 2025-06-29 04:49:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75ed0959]) (logid:3a79cdd2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:49:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4826e018]) (logid:c0a7d7ca) No inactive management server node found 2025-06-29 04:49:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4826e018]) (logid:c0a7d7ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:54,863 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d93826a8]) (logid:42b462b5) HA health check task is running... 2025-06-29 04:49:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a65a135a]) (logid:6da6558b) No inactive management server node found 2025-06-29 04:49:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a65a135a]) (logid:6da6558b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:55,403 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8784: Seq 7-8784: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:49:55,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8784 2025-06-29 04:49:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c03313b3]) (logid:706de129) No inactive management server node found 2025-06-29 04:49:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c03313b3]) (logid:706de129) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b22a9df9]) (logid:6e2a3164) Enabled non-edge zones available for scan: 1 2025-06-29 04:49:58,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b22a9df9]) (logid:6e2a3164) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:49:58,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b22a9df9]) (logid:6e2a3164) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:49:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b66d2aa1]) (logid:1cd8a5d1) Enabled non-edge zones available for scan: 1 2025-06-29 04:49:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b66d2aa1]) (logid:1cd8a5d1) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:49:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50f1117e]) (logid:6726b18f) No inactive management server node found 2025-06-29 04:49:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-50f1117e]) (logid:6726b18f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:49:58,864 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7dccbb0d]) (logid:63603846) HA health check task is running... 2025-06-29 04:49:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eb0b751]) (logid:e342deef) No inactive management server node found 2025-06-29 04:49:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eb0b751]) (logid:e342deef) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b998d7d9]) (logid:a9391a4d) No inactive management server node found 2025-06-29 04:50:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b998d7d9]) (logid:a9391a4d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91341ea3]) (logid:9ecb2ef9) No inactive management server node found 2025-06-29 04:50:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91341ea3]) (logid:9ecb2ef9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:02,865 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a3d54652]) (logid:98099f51) HA health check task is running... 2025-06-29 04:50:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1eee9e5f]) (logid:56f97a7d) No inactive management server node found 2025-06-29 04:50:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1eee9e5f]) (logid:56f97a7d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:05,405 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8785: Seq 7-8785: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:50:05,415 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8785 2025-06-29 04:50:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2d2057d]) (logid:1b55288d) No inactive management server node found 2025-06-29 04:50:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d2d2057d]) (logid:1b55288d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:06,866 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-38cb01d0]) (logid:bba3219e) HA health check task is running... 2025-06-29 04:50:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c424cb9]) (logid:b9575053) No inactive management server node found 2025-06-29 04:50:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c424cb9]) (logid:b9575053) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6f4c99d]) (logid:6c269033) No inactive management server node found 2025-06-29 04:50:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d6f4c99d]) (logid:6c269033) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbb70fe0]) (logid:258156c3) No inactive management server node found 2025-06-29 04:50:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dbb70fe0]) (logid:258156c3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:10,867 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-39b4b8f9]) (logid:e9c5614e) HA health check task is running... 2025-06-29 04:50:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cb8d83e]) (logid:2b52bf38) No inactive management server node found 2025-06-29 04:50:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4cb8d83e]) (logid:2b52bf38) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2d33bcfa]) (logid:55863306) Found 0 routers to update status. 2025-06-29 04:50:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2d33bcfa]) (logid:55863306) Found 0 VPC's to update Redundant State. 2025-06-29 04:50:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-2d33bcfa]) (logid:55863306) Found 0 networks to update RvR status. 2025-06-29 04:50:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32f6b516]) (logid:be708501) No inactive management server node found 2025-06-29 04:50:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-32f6b516]) (logid:be708501) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:14,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecabe983]) (logid:38d506b8) No inactive management server node found 2025-06-29 04:50:14,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ecabe983]) (logid:38d506b8) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:14,868 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-197c7792]) (logid:1e156cb0) HA health check task is running... 2025-06-29 04:50:15,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8786: Seq 7-8786: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:50:15,414 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8786 2025-06-29 04:50:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3b89ee8]) (logid:745cd277) No inactive management server node found 2025-06-29 04:50:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3b89ee8]) (logid:745cd277) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f4624b6]) (logid:190b9516) No inactive management server node found 2025-06-29 04:50:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3f4624b6]) (logid:190b9516) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:18,087 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-bdc02fca]) (logid:0fd37b5b) AutoScaling Monitor is running... 2025-06-29 04:50:18,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-561895e6]) (logid:ae8348f1) Resetting hosts suitable for reconnect 2025-06-29 04:50:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-561895e6]) (logid:ae8348f1) Completed resetting hosts suitable for reconnect 2025-06-29 04:50:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-561895e6]) (logid:ae8348f1) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:50:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-561895e6]) (logid:ae8348f1) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:50:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-561895e6]) (logid:ae8348f1) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:50:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-561895e6]) (logid:ae8348f1) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:50:18,167 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-3d7a8663]) (logid:0900ab2f) Starting to remove all volume_stats rows older than [Sat Jun 28 16:50:18 CEST 2025]. 2025-06-29 04:50:18,167 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-3d7a8663]) (logid:0900ab2f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:50:18 CEST 2025]. 2025-06-29 04:50:18,869 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c988fee8]) (logid:088bfd50) HA health check task is running... 2025-06-29 04:50:19,268 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-5df30626]) (logid:bfd4ab94) Starting to remove all vm_stats rows older than [Sat Jun 28 16:50:19 CEST 2025]. 2025-06-29 04:50:19,293 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-5df30626]) (logid:bfd4ab94) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:50:19 CEST 2025]. 2025-06-29 04:50:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b7ca8aa]) (logid:050a143c) No inactive management server node found 2025-06-29 04:50:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b7ca8aa]) (logid:050a143c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa2cf626]) (logid:79c5bd8b) No inactive management server node found 2025-06-29 04:50:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-aa2cf626]) (logid:79c5bd8b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:22,253 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:50:22,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:50:22,256 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) Seq 2-4725683383995203890: Routed from 90520738888109 2025-06-29 04:50:22,299 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) Seq 1-4725683383995203890: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:50:22,301 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:50:22,302 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) Seq 2-3926857400090362196: Routed from 90520738888109 2025-06-29 04:50:22,346 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-e4356c18]) (logid:bddce177) Seq 2-3926857400090362196: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:50:22,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e7d411e]) (logid:2d4fbb8d) No inactive management server node found 2025-06-29 04:50:22,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9e7d411e]) (logid:2d4fbb8d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:22,870 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c4337f2f]) (logid:9ee395d7) HA health check task is running... 2025-06-29 04:50:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6223cd5d]) (logid:4704ffbd) No inactive management server node found 2025-06-29 04:50:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6223cd5d]) (logid:4704ffbd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f41a0935]) (logid:0e9e1024) No inactive management server node found 2025-06-29 04:50:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f41a0935]) (logid:0e9e1024) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:25,404 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8787: Seq 7-8787: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:50:25,414 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8787 2025-06-29 04:50:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f918fa19]) (logid:24e50e6f) No inactive management server node found 2025-06-29 04:50:26,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f918fa19]) (logid:24e50e6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:26,871 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-0bd1dddc]) (logid:ebfc2e4c) HA health check task is running... 2025-06-29 04:50:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b5acdb53]) (logid:b1ab0146) Enabled non-edge zones available for scan: 1 2025-06-29 04:50:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b5acdb53]) (logid:b1ab0146) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:50:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b5acdb53]) (logid:b1ab0146) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:50:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0b268d17]) (logid:33b29ba5) Enabled non-edge zones available for scan: 1 2025-06-29 04:50:28,280 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-0b268d17]) (logid:33b29ba5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:50:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0da25df3]) (logid:ca65e615) No inactive management server node found 2025-06-29 04:50:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0da25df3]) (logid:ca65e615) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:29,251 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:50:29,255 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:50:29,256 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) Seq 2-4725683383995203891: Routed from 90520738888109 2025-06-29 04:50:29,311 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) Seq 1-4725683383995203891: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:50:29,345 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:50:29,346 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) Seq 2-3926857400090362197: Routed from 90520738888109 2025-06-29 04:50:29,437 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-4611fdde]) (logid:c5d56840) Seq 2-3926857400090362197: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:50:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87359fd1]) (logid:8e230272) No inactive management server node found 2025-06-29 04:50:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87359fd1]) (logid:8e230272) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:30,872 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1b6eaeef]) (logid:a5fae7a7) HA health check task is running... 2025-06-29 04:50:31,203 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) StorageCollector is running... 2025-06-29 04:50:31,209 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:50:31,209 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:50:31,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:50:31,210 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3938116399158788177: Routed from 90520738888109 2025-06-29 04:50:31,263 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 4-3938116399158788177: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:50:31,267 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:50:31,267 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:50:31,267 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:50:31,269 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3926857400090362198: Routed from 90520738888109 2025-06-29 04:50:31,280 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3926857400090362198: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:50:31,292 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:50:31,292 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:50:31,292 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:50:31,293 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3926857400090362199: Routed from 90520738888109 2025-06-29 04:50:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a482a9b]) (logid:2d4171bf) No inactive management server node found 2025-06-29 04:50:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7a482a9b]) (logid:2d4171bf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:31,386 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3926857400090362199: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:50:31,398 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:50:31,398 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:50:31,398 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:50:31,399 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-4725683383995203892: Routed from 90520738888109 2025-06-29 04:50:31,443 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 1-4725683383995203892: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:50:31,445 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:50:31,445 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:50:31,445 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:50:31,446 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3926857400090362200: Routed from 90520738888109 2025-06-29 04:50:31,490 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-b1529756]) (logid:9a486136) Seq 2-3926857400090362200: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:50:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21e2fa55]) (logid:536671ad) No inactive management server node found 2025-06-29 04:50:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21e2fa55]) (logid:536671ad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-590009c8]) (logid:4972abd9) No inactive management server node found 2025-06-29 04:50:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-590009c8]) (logid:4972abd9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:34,873 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-ea10834c]) (logid:f92a04a4) HA health check task is running... 2025-06-29 04:50:35,407 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8788: Seq 7-8788: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:50:35,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8788 2025-06-29 04:50:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39354858]) (logid:2508a47e) No inactive management server node found 2025-06-29 04:50:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39354858]) (logid:2508a47e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f6a702e]) (logid:be58106a) No inactive management server node found 2025-06-29 04:50:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f6a702e]) (logid:be58106a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc87eebf]) (logid:a4c28162) No inactive management server node found 2025-06-29 04:50:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc87eebf]) (logid:a4c28162) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:38,874 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8c420019]) (logid:28814ffc) HA health check task is running... 2025-06-29 04:50:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d730bb04]) (logid:104e1528) No inactive management server node found 2025-06-29 04:50:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d730bb04]) (logid:104e1528) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebfbd89e]) (logid:72113096) No inactive management server node found 2025-06-29 04:50:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ebfbd89e]) (logid:72113096) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:42,875 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-e750585a]) (logid:9962528e) HA health check task is running... 2025-06-29 04:50:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-2aebe0d3]) (logid:85b5ea3a) DbCollector is running... 2025-06-29 04:50:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3445f396]) (logid:de789e20) Found 0 routers to update status. 2025-06-29 04:50:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3445f396]) (logid:de789e20) Found 0 VPC's to update Redundant State. 2025-06-29 04:50:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-3445f396]) (logid:de789e20) Found 0 networks to update RvR status. 2025-06-29 04:50:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df53920a]) (logid:9da62f01) No inactive management server node found 2025-06-29 04:50:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df53920a]) (logid:9da62f01) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) ManagementServerCollector is running... 2025-06-29 04:50:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) DB properties were already loaded 2025-06-29 04:50:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:50:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Execution of process [4068003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:50:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Exit value of process [4068003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:50:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Process [4068003] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:50:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) usage install: null 2025-06-29 04:50:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Sessions active: 0 2025-06-29 04:50:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:50:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068006] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:50:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:50:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068011] for command [/bin/bash -c uname -r ]. 2025-06-29 04:50:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:50:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068012] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:50:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) system memory from /proc: 4887515136 2025-06-29 04:50:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:50:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068017] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:50:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) free memory from /proc: 349941760 2025-06-29 04:50:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:50:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068022] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:50:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) used memory from /proc: 989936 2025-06-29 04:50:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:50:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068023] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:50:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:50:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068024] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:50:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:50:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068025] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:50:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:50:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068032] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:50:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:50:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Successfully executed process [4068033] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:50:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:50:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail buffersdirect.count, 96 2025-06-29 04:50:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail buffersdirect.used, 328650 2025-06-29 04:50:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail buffersmapped.capacity, 0 2025-06-29 04:50:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail buffersmapped.count, 0 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail buffersmapped.used, 0 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail jvmuptime, 4503829 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memoryheap.committed, 671088640 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memoryheap.init, 77594624 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memoryheap.usage, 0.14399784196048476 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorynon-heap.committed, 198901760 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorynon-heap.max, -1 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorynon-heap.usage, -1.9555404E8 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorynon-heap.used, 195554040 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11872563149826713 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22030172126503816 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.Compressed-Class-Space.usage, 0.0123911052942276 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.Metaspace.usage, 0.9929464190876992 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.PS-Eden-Space.usage, 0.11355876379424329 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorytotal.committed, 869990400 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorytotal.init, 85262336 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorytotal.max, 1908932607 2025-06-29 04:50:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail memorytotal.used, 470436216 2025-06-29 04:50:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail threadsdeadlocks, [] 2025-06-29 04:50:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail threadsnew.count, 0 2025-06-29 04:50:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:50:44,130 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-a778bd48]) (logid:c62ff871) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":396870328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4503760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":469770432,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274882176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":349941760,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.38,0.23],"systemCyclesUsage":[66423665,53842897,4590338094],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:50:44,130 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 149, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":396870328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4503760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":469770432,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274882176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":349941760,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.38,0.23],"systemCyclesUsage":[66423665,53842897,4590338094],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:50:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:50:44,135 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-31:[ctx-145b1bb9]) (logid:c5180ef2) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":396870328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4503760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":469770432,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274882176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":349941760,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.38,0.23],"systemCyclesUsage":[66423665,53842897,4590338094],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:50:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:50:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 149, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:50:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":396870328,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4503760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.43,"totalInit":85262336,"totalUsed":469770432,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":274882176,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":349941760,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.43,0.38,0.23],"systemCyclesUsage":[66423665,53842897,4590338094],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:50:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94b36e0f]) (logid:14a8e8d2) No inactive management server node found 2025-06-29 04:50:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94b36e0f]) (logid:14a8e8d2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:45,405 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8789: Seq 7-8789: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:50:45,415 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8789 2025-06-29 04:50:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:50:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:50:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d14e6dc]) (logid:89ae4b4d) No inactive management server node found 2025-06-29 04:50:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d14e6dc]) (logid:89ae4b4d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:50:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:50:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:50:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:50:46,876 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-034188a0]) (logid:e68c27ce) HA health check task is running... 2025-06-29 04:50:47,095 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:50:47,096 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:50:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:50:47,100 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:50:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80e1419c]) (logid:5ff05f0d) No inactive management server node found 2025-06-29 04:50:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-80e1419c]) (logid:5ff05f0d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-887241b9]) (logid:7521546f) VM scheduler.poll is being called at 2025-06-29T02:51:00+0000 2025-06-29 04:50:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-887241b9]) (logid:7521546f) Got 0 scheduled jobs to be executed at 2025-06-29T02:51:00+0000 2025-06-29 04:50:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-887241b9]) (logid:7521546f) Cleaned up 0 VM scheduled job entries 2025-06-29 04:50:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-fff13f99]) (logid:704abcb0) VM Operation Thread Running 2025-06-29 04:50:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-fb2ae3bf]) (logid:75b0a2bc) ClusterDRS.poll is being called at 2025-06-29T02:51:00+0000 2025-06-29 04:50:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-fb2ae3bf]) (logid:75b0a2bc) Removed 0 old drs migration plans 2025-06-29 04:50:49,373 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee9c319e]) (logid:6ca62b9b) No inactive management server node found 2025-06-29 04:50:49,373 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee9c319e]) (logid:6ca62b9b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76884ff9]) (logid:9177ccde) No inactive management server node found 2025-06-29 04:50:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76884ff9]) (logid:9177ccde) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:50,877 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c7d08ec5]) (logid:4991c0ed) HA health check task is running... 2025-06-29 04:50:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9165259b]) (logid:005c1d2f) No inactive management server node found 2025-06-29 04:50:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9165259b]) (logid:005c1d2f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0682213]) (logid:d68d4307) No inactive management server node found 2025-06-29 04:50:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0682213]) (logid:d68d4307) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:54,878 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-8573b545]) (logid:2c92db3b) HA health check task is running... 2025-06-29 04:50:55,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c5ffb5f]) (logid:ee4d07c1) No inactive management server node found 2025-06-29 04:50:55,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c5ffb5f]) (logid:ee4d07c1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:55,407 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8791: Seq 7-8791: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:50:55,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8791 2025-06-29 04:50:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7063a3ca]) (logid:f34fa017) No inactive management server node found 2025-06-29 04:50:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7063a3ca]) (logid:f34fa017) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b03ff55d]) (logid:86b39ae3) Enabled non-edge zones available for scan: 1 2025-06-29 04:50:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b03ff55d]) (logid:86b39ae3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:50:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b03ff55d]) (logid:86b39ae3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:50:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-15c5de52]) (logid:8f8fe9f4) Enabled non-edge zones available for scan: 1 2025-06-29 04:50:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-15c5de52]) (logid:8f8fe9f4) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:50:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaeda529]) (logid:f31652d4) No inactive management server node found 2025-06-29 04:50:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eaeda529]) (logid:f31652d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:50:58,879 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-f88c88eb]) (logid:d9195749) HA health check task is running... 2025-06-29 04:50:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-826230c9]) (logid:eceeb4e5) No inactive management server node found 2025-06-29 04:50:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-826230c9]) (logid:eceeb4e5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97f08871]) (logid:0839492d) No inactive management server node found 2025-06-29 04:51:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-97f08871]) (logid:0839492d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06dc0ee6]) (logid:1af291c0) No inactive management server node found 2025-06-29 04:51:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-06dc0ee6]) (logid:1af291c0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:02,881 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-dde3ce01]) (logid:f5140c51) HA health check task is running... 2025-06-29 04:51:04,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab70356e]) (logid:3947ea60) No inactive management server node found 2025-06-29 04:51:04,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab70356e]) (logid:3947ea60) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:05,411 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8792: Seq 7-8792: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:51:05,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8792 2025-06-29 04:51:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39c98507]) (logid:ed4a01c0) No inactive management server node found 2025-06-29 04:51:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39c98507]) (logid:ed4a01c0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:06,881 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-34ad2452]) (logid:9a572472) HA health check task is running... 2025-06-29 04:51:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87cc16b7]) (logid:873eb5be) No inactive management server node found 2025-06-29 04:51:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-87cc16b7]) (logid:873eb5be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f8b7d8]) (logid:ac572bd6) No inactive management server node found 2025-06-29 04:51:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-93f8b7d8]) (logid:ac572bd6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4a7e86a]) (logid:8a569b2b) No inactive management server node found 2025-06-29 04:51:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b4a7e86a]) (logid:8a569b2b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:10,882 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-00e41883]) (logid:08ccb0ff) HA health check task is running... 2025-06-29 04:51:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dad90972]) (logid:1906e226) No inactive management server node found 2025-06-29 04:51:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dad90972]) (logid:1906e226) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8750b6fc]) (logid:f1142a80) Found 0 routers to update status. 2025-06-29 04:51:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8750b6fc]) (logid:f1142a80) Found 0 VPC's to update Redundant State. 2025-06-29 04:51:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8750b6fc]) (logid:f1142a80) Found 0 networks to update RvR status. 2025-06-29 04:51:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-621d77b4]) (logid:379d71e9) No inactive management server node found 2025-06-29 04:51:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-621d77b4]) (logid:379d71e9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b5edf3e]) (logid:fb101ded) No inactive management server node found 2025-06-29 04:51:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b5edf3e]) (logid:fb101ded) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:14,884 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-083baaaf]) (logid:8e77bc41) HA health check task is running... 2025-06-29 04:51:15,406 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8793: Seq 7-8793: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:51:15,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8793 2025-06-29 04:51:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bc13211]) (logid:ed1e10f4) No inactive management server node found 2025-06-29 04:51:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bc13211]) (logid:ed1e10f4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffa18903]) (logid:9fca1bf1) No inactive management server node found 2025-06-29 04:51:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ffa18903]) (logid:9fca1bf1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:18,089 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-d3f6e1e1]) (logid:d3d7cc01) AutoScaling Monitor is running... 2025-06-29 04:51:18,169 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-6b8a5504]) (logid:5b822edb) Starting to remove all volume_stats rows older than [Sat Jun 28 16:51:18 CEST 2025]. 2025-06-29 04:51:18,170 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-6b8a5504]) (logid:5b822edb) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:51:18 CEST 2025]. 2025-06-29 04:51:18,885 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-fe606f4f]) (logid:3f7f0d9c) HA health check task is running... 2025-06-29 04:51:19,294 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-7cefc05d]) (logid:0360fdd9) Starting to remove all vm_stats rows older than [Sat Jun 28 16:51:19 CEST 2025]. 2025-06-29 04:51:19,327 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-7cefc05d]) (logid:0360fdd9) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:51:19 CEST 2025]. 2025-06-29 04:51:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84e4fce0]) (logid:de4af538) No inactive management server node found 2025-06-29 04:51:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84e4fce0]) (logid:de4af538) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99831409]) (logid:effbdb0c) No inactive management server node found 2025-06-29 04:51:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-99831409]) (logid:effbdb0c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:22,353 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:51:22,356 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:51:22,358 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) Seq 2-4725683383995203893: Routed from 90520738888109 2025-06-29 04:51:22,364 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d2fb13b]) (logid:e0a6cfea) No inactive management server node found 2025-06-29 04:51:22,364 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6d2fb13b]) (logid:e0a6cfea) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:22,402 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) Seq 1-4725683383995203893: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:51:22,403 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:51:22,405 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) Seq 2-3926857400090362201: Routed from 90520738888109 2025-06-29 04:51:22,449 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-fcdfa43e]) (logid:d4fd80f5) Seq 2-3926857400090362201: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:51:22,886 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5811d929]) (logid:5ed20f64) HA health check task is running... 2025-06-29 04:51:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0850d3f]) (logid:dddeefad) No inactive management server node found 2025-06-29 04:51:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0850d3f]) (logid:dddeefad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c55473e1]) (logid:6d15bffc) No inactive management server node found 2025-06-29 04:51:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c55473e1]) (logid:6d15bffc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:25,408 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8794: Seq 7-8794: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:51:25,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8794 2025-06-29 04:51:26,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d42ce86d]) (logid:14b35de7) No inactive management server node found 2025-06-29 04:51:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d42ce86d]) (logid:14b35de7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:26,887 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-79eb957c]) (logid:776f865e) HA health check task is running... 2025-06-29 04:51:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-acee9570]) (logid:55ae2f50) Enabled non-edge zones available for scan: 1 2025-06-29 04:51:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-acee9570]) (logid:55ae2f50) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:51:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-acee9570]) (logid:55ae2f50) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:51:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c8462c0e]) (logid:11c38391) Enabled non-edge zones available for scan: 1 2025-06-29 04:51:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c8462c0e]) (logid:11c38391) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:51:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d91c195e]) (logid:45308e44) No inactive management server node found 2025-06-29 04:51:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d91c195e]) (logid:45308e44) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:29,496 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:51:29,499 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:51:29,500 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) Seq 2-4725683383995203894: Routed from 90520738888109 2025-06-29 04:51:29,554 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) Seq 1-4725683383995203894: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:51:29,587 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:51:29,588 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) Seq 2-3926857400090362202: Routed from 90520738888109 2025-06-29 04:51:29,677 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-18a1379d]) (logid:5d83467c) Seq 2-3926857400090362202: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:51:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dd32b6b]) (logid:b4791b95) No inactive management server node found 2025-06-29 04:51:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6dd32b6b]) (logid:b4791b95) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:30,407 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8795: Seq 7-8795: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:51:30,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8795 2025-06-29 04:51:30,889 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-c8577fc5]) (logid:c01eff9a) HA health check task is running... 2025-06-29 04:51:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-430bee27]) (logid:f93c72d8) No inactive management server node found 2025-06-29 04:51:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-430bee27]) (logid:f93c72d8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:31,490 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) StorageCollector is running... 2025-06-29 04:51:31,498 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:51:31,499 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:51:31,499 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:51:31,499 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-3938116399158788178: Routed from 90520738888109 2025-06-29 04:51:31,553 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 4-3938116399158788178: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:51:31,558 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:51:31,558 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:51:31,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:51:31,559 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-4725683383995203895: Routed from 90520738888109 2025-06-29 04:51:31,609 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 1-4725683383995203895: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:51:31,620 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:51:31,620 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:51:31,620 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:51:31,621 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-3926857400090362203: Routed from 90520738888109 2025-06-29 04:51:31,632 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-3926857400090362203: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:51:31,641 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:51:31,641 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:51:31,641 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:51:31,642 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-4725683383995203896: Routed from 90520738888109 2025-06-29 04:51:31,726 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 1-4725683383995203896: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:51:31,728 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:51:31,728 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:51:31,729 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:51:31,730 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-3926857400090362204: Routed from 90520738888109 2025-06-29 04:51:31,775 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-338790a2]) (logid:15f82c68) Seq 2-3926857400090362204: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:51:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfe2e8f6]) (logid:458d489e) No inactive management server node found 2025-06-29 04:51:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cfe2e8f6]) (logid:458d489e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77ead66d]) (logid:d9373569) No inactive management server node found 2025-06-29 04:51:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77ead66d]) (logid:d9373569) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:34,890 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-ab87664b]) (logid:9a1124a5) HA health check task is running... 2025-06-29 04:51:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee2f04b8]) (logid:a3095c3b) No inactive management server node found 2025-06-29 04:51:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ee2f04b8]) (logid:a3095c3b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04edcf34]) (logid:9e9f5342) No inactive management server node found 2025-06-29 04:51:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-04edcf34]) (logid:9e9f5342) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9397d5ae]) (logid:093b44fb) No inactive management server node found 2025-06-29 04:51:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9397d5ae]) (logid:093b44fb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:38,891 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-345fd45f]) (logid:6908627c) HA health check task is running... 2025-06-29 04:51:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-711a77fa]) (logid:688534d3) No inactive management server node found 2025-06-29 04:51:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-711a77fa]) (logid:688534d3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:40,408 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8796: Seq 7-8796: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:51:40,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8796 2025-06-29 04:51:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-511c87db]) (logid:12910b1e) No inactive management server node found 2025-06-29 04:51:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-511c87db]) (logid:12910b1e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:42,892 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b0de9cc4]) (logid:ae813821) HA health check task is running... 2025-06-29 04:51:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-566f4505]) (logid:c78cd918) DbCollector is running... 2025-06-29 04:51:43,096 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-923170fc]) (logid:78d91d46) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:51:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-47e16017]) (logid:02928c54) Found 0 running routers. 2025-06-29 04:51:43,098 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-923170fc]) (logid:78d91d46) Snapshot scheduler is being called at [2025-06-29T02:51:43+0000]. 2025-06-29 04:51:43,099 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-923170fc]) (logid:78d91d46) There are [0] scheduled snapshots to be executed at [2025-06-29T02:51:43+0000]. 2025-06-29 04:51:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c80eb5d4]) (logid:cba90028) Found 0 routers to update status. 2025-06-29 04:51:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c80eb5d4]) (logid:cba90028) Found 0 VPC's to update Redundant State. 2025-06-29 04:51:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-c80eb5d4]) (logid:cba90028) Found 0 networks to update RvR status. 2025-06-29 04:51:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-f933f799]) (logid:891c0804) External devices are not used. Skipping external device usage collection 2025-06-29 04:51:43,139 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-3c16cf53]) (logid:3acadd0b) Started resource counters recalculation periodic task. 2025-06-29 04:51:43,142 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-3c16cf53]) (logid:3acadd0b) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:51:43,307 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-161cf268]) (logid:2b1bef24) Backup backup.poll is being called at 2025-06-29T02:51:43+0000 2025-06-29 04:51:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9cf969b]) (logid:10738d6a) No inactive management server node found 2025-06-29 04:51:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9cf969b]) (logid:10738d6a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:43,538 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-3c16cf53]) (logid:3acadd0b) Finished resource counters recalculation periodic task. 2025-06-29 04:51:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) ManagementServerCollector is running... 2025-06-29 04:51:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) DB properties were already loaded 2025-06-29 04:51:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:51:44,056 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Execution of process [4068044] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:51:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Exit value of process [4068044] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:51:44,056 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Process [4068044] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:51:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) usage install: null 2025-06-29 04:51:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Sessions active: 0 2025-06-29 04:51:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:51:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068047] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:51:44,065 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:51:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068052] for command [/bin/bash -c uname -r ]. 2025-06-29 04:51:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:51:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068053] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:51:44,076 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) system memory from /proc: 4887515136 2025-06-29 04:51:44,076 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:51:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068058] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:51:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) free memory from /proc: 351305728 2025-06-29 04:51:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:51:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068063] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:51:44,093 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) used memory from /proc: 989936 2025-06-29 04:51:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:51:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068064] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:51:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:51:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068065] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:51:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:51:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068066] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:51:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:51:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068073] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:51:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:51:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Successfully executed process [4068074] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail buffersdirect.count, 96 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail buffersdirect.used, 328650 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail buffersmapped.capacity, 0 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail buffersmapped.count, 0 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail buffersmapped.used, 0 2025-06-29 04:51:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail jvmuptime, 4563831 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memoryheap.committed, 671088640 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memoryheap.init, 77594624 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memoryheap.usage, 0.16062020770929175 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorynon-heap.committed, 198901760 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorynon-heap.max, -1 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorynon-heap.usage, -1.95597112E8 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorynon-heap.used, 195597112 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11888912956544921 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2204516862732029 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.Compressed-Class-Space.usage, 0.0123911052942276 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.Metaspace.usage, 0.9929789253875118 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.PS-Eden-Space.usage, 0.16308690591785752 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorytotal.committed, 869990400 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorytotal.init, 85262336 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorytotal.max, 1908932607 2025-06-29 04:51:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail memorytotal.used, 502392744 2025-06-29 04:51:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail threadsdeadlocks, [] 2025-06-29 04:51:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail threadsnew.count, 0 2025-06-29 04:51:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:51:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-9ef26242]) (logid:49ea5619) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":365315944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4563759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":501368912,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":306613152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351305728,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.35,0.23],"systemCyclesUsage":[66424003,53843197,4590361205],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:51:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 151, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":365315944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4563759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":501368912,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":306613152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351305728,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.35,0.23],"systemCyclesUsage":[66424003,53843197,4590361205],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:51:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:51:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:51:44,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 151, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":365315944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4563759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":501368912,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":306613152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351305728,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.35,0.23],"systemCyclesUsage":[66424003,53843197,4590361205],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:51:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-32:[ctx-f503ccd7]) (logid:fa2e53bd) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:51:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":365315944,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4563759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.27,"totalInit":85262336,"totalUsed":501368912,"totalCommitted":869990400,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":306613152,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":351305728,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.27,0.35,0.23],"systemCyclesUsage":[66424003,53843197,4590361205],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:51:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb052ac0]) (logid:4032dfc4) No inactive management server node found 2025-06-29 04:51:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fb052ac0]) (logid:4032dfc4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:51:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:51:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec8c06df]) (logid:a4799ff0) No inactive management server node found 2025-06-29 04:51:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec8c06df]) (logid:a4799ff0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:46,641 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:51:46,643 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:51:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:51:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:51:46,893 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c8cbd172]) (logid:7c4f48bf) HA health check task is running... 2025-06-29 04:51:47,077 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:51:47,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:51:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:51:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:51:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e9226ce]) (logid:a53d83a3) No inactive management server node found 2025-06-29 04:51:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e9226ce]) (logid:a53d83a3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:48,061 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-1:[ctx-ee793e84]) (logid:2fc69d37) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:51:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84f0d4dd]) (logid:beb7265f) Resetting hosts suitable for reconnect 2025-06-29 04:51:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84f0d4dd]) (logid:beb7265f) Completed resetting hosts suitable for reconnect 2025-06-29 04:51:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84f0d4dd]) (logid:beb7265f) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:51:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84f0d4dd]) (logid:beb7265f) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:51:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84f0d4dd]) (logid:beb7265f) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:51:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-84f0d4dd]) (logid:beb7265f) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:51:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3bf4c706]) (logid:3203c1b0) VM scheduler.poll is being called at 2025-06-29T02:52:00+0000 2025-06-29 04:51:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3bf4c706]) (logid:3203c1b0) Got 0 scheduled jobs to be executed at 2025-06-29T02:52:00+0000 2025-06-29 04:51:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-3bf4c706]) (logid:3203c1b0) Cleaned up 0 VM scheduled job entries 2025-06-29 04:51:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-71cc02cb]) (logid:56789df6) VM Operation Thread Running 2025-06-29 04:51:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e04ce1f0]) (logid:a20ff0ca) ClusterDRS.poll is being called at 2025-06-29T02:52:00+0000 2025-06-29 04:51:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e04ce1f0]) (logid:a20ff0ca) Removed 0 old drs migration plans 2025-06-29 04:51:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54ae06c0]) (logid:fa612ad9) No inactive management server node found 2025-06-29 04:51:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-54ae06c0]) (logid:fa612ad9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:50,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8798: Seq 7-8798: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:51:50,419 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8798 2025-06-29 04:51:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a817a280]) (logid:029049c3) No inactive management server node found 2025-06-29 04:51:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a817a280]) (logid:029049c3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:50,894 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-43a1fd4c]) (logid:a6dffba4) HA health check task is running... 2025-06-29 04:51:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4bc6353]) (logid:b60ed3bc) No inactive management server node found 2025-06-29 04:51:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4bc6353]) (logid:b60ed3bc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edf024a6]) (logid:5944f2f2) No inactive management server node found 2025-06-29 04:51:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-edf024a6]) (logid:5944f2f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:54,896 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-82420d5b]) (logid:2e4a044c) HA health check task is running... 2025-06-29 04:51:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fa05a1]) (logid:16f8c4c6) No inactive management server node found 2025-06-29 04:51:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5fa05a1]) (logid:16f8c4c6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae85bf92]) (logid:f1be9f6e) No inactive management server node found 2025-06-29 04:51:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ae85bf92]) (logid:f1be9f6e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f9eb8d6c]) (logid:aa2b844a) Enabled non-edge zones available for scan: 1 2025-06-29 04:51:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-f9eb8d6c]) (logid:aa2b844a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:51:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-f9eb8d6c]) (logid:aa2b844a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:51:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c6363af8]) (logid:b3516ea9) Enabled non-edge zones available for scan: 1 2025-06-29 04:51:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c6363af8]) (logid:b3516ea9) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:51:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c4d97ba]) (logid:9beb57c6) No inactive management server node found 2025-06-29 04:51:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2c4d97ba]) (logid:9beb57c6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:51:58,897 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-6fa1d641]) (logid:c263166e) HA health check task is running... 2025-06-29 04:51:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f3f217a]) (logid:1cc2b7d2) No inactive management server node found 2025-06-29 04:51:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f3f217a]) (logid:1cc2b7d2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:00,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8799: Seq 7-8799: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:00,419 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8799 2025-06-29 04:52:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4829ca8e]) (logid:3320e8cb) No inactive management server node found 2025-06-29 04:52:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4829ca8e]) (logid:3320e8cb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf705fae]) (logid:d2ab42b9) No inactive management server node found 2025-06-29 04:52:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf705fae]) (logid:d2ab42b9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:02,898 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-318deac4]) (logid:81a7ec69) HA health check task is running... 2025-06-29 04:52:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4039fe59]) (logid:18bf5273) No inactive management server node found 2025-06-29 04:52:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4039fe59]) (logid:18bf5273) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ed7b489]) (logid:310517b4) No inactive management server node found 2025-06-29 04:52:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9ed7b489]) (logid:310517b4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:06,899 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-c2687f5e]) (logid:477c47c2) HA health check task is running... 2025-06-29 04:52:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ec01874]) (logid:6d6542ba) No inactive management server node found 2025-06-29 04:52:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1ec01874]) (logid:6d6542ba) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89433391]) (logid:bc897b83) No inactive management server node found 2025-06-29 04:52:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89433391]) (logid:bc897b83) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-90685f61]) (logid:3f983fb1) No inactive management server node found 2025-06-29 04:52:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-90685f61]) (logid:3f983fb1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:10,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8800: Seq 7-8800: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:10,419 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8800 2025-06-29 04:52:10,901 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-076629da]) (logid:ac71c76f) HA health check task is running... 2025-06-29 04:52:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adcaaf2f]) (logid:a3d6ac78) No inactive management server node found 2025-06-29 04:52:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-adcaaf2f]) (logid:a3d6ac78) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-204b3e66]) (logid:76a46f8d) Checking if any host reservation can be released ... 2025-06-29 04:52:13,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-204b3e66]) (logid:76a46f8d) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:13,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-204b3e66]) (logid:76a46f8d) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:13,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-204b3e66]) (logid:76a46f8d) Done running HostReservationReleaseChecker ... 2025-06-29 04:52:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2c7eb0b]) (logid:5fe56550) Found 0 routers to update status. 2025-06-29 04:52:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2c7eb0b]) (logid:5fe56550) Found 0 VPC's to update Redundant State. 2025-06-29 04:52:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b2c7eb0b]) (logid:5fe56550) Found 0 networks to update RvR status. 2025-06-29 04:52:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Running Capacity Checker ... 2025-06-29 04:52:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) recalculating system capacity 2025-06-29 04:52:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Executing cpu/ram capacity update 2025-06-29 04:52:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:13,173 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:13,183 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:13,187 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:52:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:52:13,188 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:52:13,200 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:13,201 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:13,207 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:52:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:52:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-42-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:52:13,218 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Done executing cpu/ram capacity update 2025-06-29 04:52:13,218 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Executing storage capacity update 2025-06-29 04:52:13,225 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:52:13,226 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:52:13,226 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:52:13,229 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:52:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:52:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:52:13,235 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:52:13,238 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:52:13,244 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:52:13,245 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:52:13,245 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:52:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:52:13,256 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:52:13,257 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:52:13,257 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:52:13,261 DEBUG [c.c.s.StorageManagerImpl] (pool-43-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:52:13,262 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Done executing storage capacity update 2025-06-29 04:52:13,262 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Executing capacity updates for public ip and Vlans 2025-06-29 04:52:13,278 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Done capacity updates for public ip and Vlans 2025-06-29 04:52:13,278 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Executing capacity updates for private ip 2025-06-29 04:52:13,281 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Done executing capacity updates for private ip 2025-06-29 04:52:13,281 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Done recalculating system capacity 2025-06-29 04:52:13,316 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-01f77473]) (logid:6555c087) Done running Capacity Checker ... 2025-06-29 04:52:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc861297]) (logid:a7f64453) No inactive management server node found 2025-06-29 04:52:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc861297]) (logid:a7f64453) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39ee0c74]) (logid:e6d90ff2) No inactive management server node found 2025-06-29 04:52:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39ee0c74]) (logid:e6d90ff2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:14,902 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-af240465]) (logid:49ab1cbf) HA health check task is running... 2025-06-29 04:52:15,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8801: Seq 7-8801: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:15,419 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8801 2025-06-29 04:52:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f40709a3]) (logid:47f250c2) No inactive management server node found 2025-06-29 04:52:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f40709a3]) (logid:47f250c2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6cb0fc8f]) (logid:88231cc3) No inactive management server node found 2025-06-29 04:52:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6cb0fc8f]) (logid:88231cc3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:52:18,091 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-1baa0dab]) (logid:bf1e91ec) AutoScaling Monitor is running... 2025-06-29 04:52:18,172 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b5afef0b]) (logid:d8ee382c) Starting to remove all volume_stats rows older than [Sat Jun 28 16:52:18 CEST 2025]. 2025-06-29 04:52:18,172 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-b5afef0b]) (logid:d8ee382c) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:52:18 CEST 2025]. 2025-06-29 04:52:18,903 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-653c5041]) (logid:6cce08f2) HA health check task is running... 2025-06-29 04:52:19,329 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-ada4e33d]) (logid:27d87a24) Starting to remove all vm_stats rows older than [Sat Jun 28 16:52:19 CEST 2025]. 2025-06-29 04:52:19,355 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-ada4e33d]) (logid:27d87a24) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:52:19 CEST 2025]. 2025-06-29 04:52:19,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db81f093]) (logid:257a69e3) No inactive management server node found 2025-06-29 04:52:19,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-db81f093]) (logid:257a69e3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bfc1e1e]) (logid:cb6b1406) No inactive management server node found 2025-06-29 04:52:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8bfc1e1e]) (logid:cb6b1406) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d04369a]) (logid:975d522d) No inactive management server node found 2025-06-29 04:52:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4d04369a]) (logid:975d522d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:22,451 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:52:22,453 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:52:22,455 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) Seq 2-4725683383995203897: Routed from 90520738888109 2025-06-29 04:52:22,498 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) Seq 1-4725683383995203897: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:52:22,499 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:52:22,500 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) Seq 2-3926857400090362205: Routed from 90520738888109 2025-06-29 04:52:22,544 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-d8a0c392]) (logid:41f9ee9a) Seq 2-3926857400090362205: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:52:22,904 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-1fa75293]) (logid:90fdf79d) HA health check task is running... 2025-06-29 04:52:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fe34ad3]) (logid:6a93c95c) No inactive management server node found 2025-06-29 04:52:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5fe34ad3]) (logid:6a93c95c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:25,410 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8802: Seq 7-8802: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:25,440 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d845863]) (logid:c46bc2ca) No inactive management server node found 2025-06-29 04:52:25,440 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d845863]) (logid:c46bc2ca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:25,440 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8802 2025-06-29 04:52:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad3f808a]) (logid:485abd88) No inactive management server node found 2025-06-29 04:52:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad3f808a]) (logid:485abd88) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:26,906 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b5e68a22]) (logid:71ee1f0d) HA health check task is running... 2025-06-29 04:52:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4a073298]) (logid:753834f6) Enabled non-edge zones available for scan: 1 2025-06-29 04:52:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-4a073298]) (logid:753834f6) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:52:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4a073298]) (logid:753834f6) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:52:28,270 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5add0417]) (logid:6bc91591) Enabled non-edge zones available for scan: 1 2025-06-29 04:52:28,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5add0417]) (logid:6bc91591) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:52:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18029952]) (logid:eec265fd) No inactive management server node found 2025-06-29 04:52:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-18029952]) (logid:eec265fd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:29,732 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:52:29,735 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:52:29,735 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) Seq 2-4725683383995203898: Routed from 90520738888109 2025-06-29 04:52:29,788 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) Seq 1-4725683383995203898: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:52:29,823 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:52:29,824 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) Seq 2-3926857400090362206: Routed from 90520738888109 2025-06-29 04:52:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94636813]) (logid:47ee09db) No inactive management server node found 2025-06-29 04:52:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-94636813]) (logid:47ee09db) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:29,910 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6cb51a98]) (logid:3c03758d) Seq 2-3926857400090362206: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:52:30,907 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-98c04027]) (logid:c49a2236) HA health check task is running... 2025-06-29 04:52:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89e51634]) (logid:c8ff76bb) No inactive management server node found 2025-06-29 04:52:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89e51634]) (logid:c8ff76bb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:31,775 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) StorageCollector is running... 2025-06-29 04:52:31,780 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:52:31,780 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:52:31,780 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:52:31,780 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3938116399158788179: Routed from 90520738888109 2025-06-29 04:52:31,834 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 4-3938116399158788179: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:52:31,837 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:52:31,837 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:52:31,837 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:52:31,838 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3926857400090362207: Routed from 90520738888109 2025-06-29 04:52:31,849 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3926857400090362207: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:52:31,859 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:52:31,859 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:52:31,859 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:52:31,860 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3926857400090362208: Routed from 90520738888109 2025-06-29 04:52:31,952 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3926857400090362208: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:52:31,962 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:52:31,962 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:52:31,962 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:52:31,963 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-4725683383995203899: Routed from 90520738888109 2025-06-29 04:52:32,012 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 1-4725683383995203899: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:52:32,014 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:52:32,014 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:52:32,014 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:52:32,016 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3926857400090362209: Routed from 90520738888109 2025-06-29 04:52:32,061 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-b93d4ec3]) (logid:5684b888) Seq 2-3926857400090362209: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:52:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fca5dbe4]) (logid:44962791) No inactive management server node found 2025-06-29 04:52:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fca5dbe4]) (logid:44962791) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3baf4e98]) (logid:4897855a) No inactive management server node found 2025-06-29 04:52:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3baf4e98]) (logid:4897855a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:34,908 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5ae6a1a2]) (logid:c1ab09fc) HA health check task is running... 2025-06-29 04:52:35,411 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8803: Seq 7-8803: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:35,422 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8803 2025-06-29 04:52:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08ba5b7b]) (logid:d671da23) No inactive management server node found 2025-06-29 04:52:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08ba5b7b]) (logid:d671da23) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5531b18]) (logid:2dba4a1a) No inactive management server node found 2025-06-29 04:52:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5531b18]) (logid:2dba4a1a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:38,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60eae1d8]) (logid:1c3d69a3) No inactive management server node found 2025-06-29 04:52:38,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60eae1d8]) (logid:1c3d69a3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:38,909 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-5b556b7b]) (logid:fd6b6f66) HA health check task is running... 2025-06-29 04:52:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-663fd24b]) (logid:528ab5b7) No inactive management server node found 2025-06-29 04:52:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-663fd24b]) (logid:528ab5b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98f85f9c]) (logid:59693ffa) No inactive management server node found 2025-06-29 04:52:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98f85f9c]) (logid:59693ffa) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:42,910 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8ae0a2b4]) (logid:411eaf25) HA health check task is running... 2025-06-29 04:52:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-856664d8]) (logid:8d55bec9) DbCollector is running... 2025-06-29 04:52:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-411f6353]) (logid:6141997c) Found 0 routers to update status. 2025-06-29 04:52:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-411f6353]) (logid:6141997c) Found 0 VPC's to update Redundant State. 2025-06-29 04:52:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-411f6353]) (logid:6141997c) Found 0 networks to update RvR status. 2025-06-29 04:52:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f6def14]) (logid:d842d498) No inactive management server node found 2025-06-29 04:52:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f6def14]) (logid:d842d498) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) ManagementServerCollector is running... 2025-06-29 04:52:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) DB properties were already loaded 2025-06-29 04:52:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:52:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Execution of process [4068085] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:52:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Exit value of process [4068085] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:52:44,056 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Process [4068085] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:52:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) usage install: null 2025-06-29 04:52:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Sessions active: 0 2025-06-29 04:52:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:52:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068088] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:52:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:52:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068093] for command [/bin/bash -c uname -r ]. 2025-06-29 04:52:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:52:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068094] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:52:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) system memory from /proc: 4887515136 2025-06-29 04:52:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:52:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068099] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:52:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) free memory from /proc: 345432064 2025-06-29 04:52:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:52:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068104] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:52:44,090 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) used memory from /proc: 989936 2025-06-29 04:52:44,090 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:52:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068105] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:52:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:52:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068106] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:52:44,107 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:52:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068107] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:52:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:52:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068114] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:52:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:52:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Successfully executed process [4068115] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail buffersdirect.count, 96 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail buffersdirect.used, 328650 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail buffersmapped.capacity, 0 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail buffersmapped.count, 0 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail buffersmapped.used, 0 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail jvmuptime, 4623829 2025-06-29 04:52:44,125 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memoryheap.committed, 671088640 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memoryheap.init, 77594624 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memoryheap.usage, 0.17675876800780177 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorynon-heap.committed, 199032832 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorynon-heap.max, -1 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorynon-heap.usage, -1.9569264E8 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorynon-heap.used, 195692640 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.1191224006931485 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22060998267071016 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.Compressed-Class-Space.usage, 0.012398816645145416 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.Metaspace.usage, 0.992793332558429 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.PS-Eden-Space.usage, 0.21061533965940296 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorytotal.committed, 870121472 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorytotal.init, 85262336 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorytotal.max, 1908932607 2025-06-29 04:52:44,126 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail memorytotal.used, 533113216 2025-06-29 04:52:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail threadsdeadlocks, [] 2025-06-29 04:52:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail threadsnew.count, 0 2025-06-29 04:52:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:52:44,129 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-a0646000]) (logid:b51f3a12) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":334476744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4623759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":532304536,"totalCommitted":870121472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":337420576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":345432064,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.28,0.21],"systemCyclesUsage":[66424330,53843500,4590384334],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:52:44,130 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 153, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":334476744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4623759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":532304536,"totalCommitted":870121472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":337420576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":345432064,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.28,0.21],"systemCyclesUsage":[66424330,53843500,4590384334],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:52:44,130 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:52:44,135 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-32:[ctx-3ab6d452]) (logid:840140a6) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":334476744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4623759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":532304536,"totalCommitted":870121472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":337420576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":345432064,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.28,0.21],"systemCyclesUsage":[66424330,53843500,4590384334],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:52:44,135 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:52:44,135 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 153, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:52:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":334476744,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4623759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":532304536,"totalCommitted":870121472,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":337420576,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":345432064,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.28,0.21],"systemCyclesUsage":[66424330,53843500,4590384334],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:52:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65458b11]) (logid:6319c428) No inactive management server node found 2025-06-29 04:52:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65458b11]) (logid:6319c428) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:45,411 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8804: Seq 7-8804: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:45,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8804 2025-06-29 04:52:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:52:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:52:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dae9a979]) (logid:6b4f96e9) No inactive management server node found 2025-06-29 04:52:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dae9a979]) (logid:6b4f96e9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:52:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:52:46,911 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-be027798]) (logid:85927442) HA health check task is running... 2025-06-29 04:52:47,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:52:47,102 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:52:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14c4a458]) (logid:353451f0) No inactive management server node found 2025-06-29 04:52:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-14c4a458]) (logid:353451f0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:52:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-4411826b]) (logid:9a7a6593) VM scheduler.poll is being called at 2025-06-29T02:53:00+0000 2025-06-29 04:52:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-4411826b]) (logid:9a7a6593) Got 0 scheduled jobs to be executed at 2025-06-29T02:53:00+0000 2025-06-29 04:52:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-4411826b]) (logid:9a7a6593) Cleaned up 0 VM scheduled job entries 2025-06-29 04:52:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-0a5a8aca]) (logid:28e669a6) VM Operation Thread Running 2025-06-29 04:52:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-72a3785c]) (logid:6c8eafb8) ClusterDRS.poll is being called at 2025-06-29T02:53:00+0000 2025-06-29 04:52:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-72a3785c]) (logid:6c8eafb8) Removed 0 old drs migration plans 2025-06-29 04:52:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc931588]) (logid:d5620fca) No inactive management server node found 2025-06-29 04:52:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dc931588]) (logid:d5620fca) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08a12a4e]) (logid:04580eaf) No inactive management server node found 2025-06-29 04:52:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-08a12a4e]) (logid:04580eaf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:50,912 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d6b1ace1]) (logid:b09da3e3) HA health check task is running... 2025-06-29 04:52:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0c1024e]) (logid:dc9426b0) No inactive management server node found 2025-06-29 04:52:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0c1024e]) (logid:dc9426b0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:53,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2426f60f]) (logid:03af01ec) No inactive management server node found 2025-06-29 04:52:53,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2426f60f]) (logid:03af01ec) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:54,914 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-7f4b408a]) (logid:594101f2) HA health check task is running... 2025-06-29 04:52:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c5ad02b]) (logid:edf8b3e4) No inactive management server node found 2025-06-29 04:52:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9c5ad02b]) (logid:edf8b3e4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:55,411 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8806: Seq 7-8806: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:52:55,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8806 2025-06-29 04:52:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21def0e1]) (logid:c4517daf) No inactive management server node found 2025-06-29 04:52:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-21def0e1]) (logid:c4517daf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6b1ff4ca]) (logid:9f11adf6) Enabled non-edge zones available for scan: 1 2025-06-29 04:52:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6b1ff4ca]) (logid:9f11adf6) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:52:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6b1ff4ca]) (logid:9f11adf6) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:52:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1eddf241]) (logid:2d755d11) Enabled non-edge zones available for scan: 1 2025-06-29 04:52:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-1eddf241]) (logid:2d755d11) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:52:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3a41805]) (logid:f8ff7ee8) No inactive management server node found 2025-06-29 04:52:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d3a41805]) (logid:f8ff7ee8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:52:58,915 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-85273d57]) (logid:7b6eb4c3) HA health check task is running... 2025-06-29 04:52:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7e23243]) (logid:1f224229) No inactive management server node found 2025-06-29 04:52:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c7e23243]) (logid:1f224229) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea3b2752]) (logid:c99f78ce) No inactive management server node found 2025-06-29 04:53:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea3b2752]) (logid:c99f78ce) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a37fea15]) (logid:f4639731) No inactive management server node found 2025-06-29 04:53:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a37fea15]) (logid:f4639731) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:02,916 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1a686ee2]) (logid:2b78b240) HA health check task is running... 2025-06-29 04:53:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec75264a]) (logid:a4aa5513) No inactive management server node found 2025-06-29 04:53:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ec75264a]) (logid:a4aa5513) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:05,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8807: Seq 7-8807: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:53:05,436 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8807 2025-06-29 04:53:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a267fa03]) (logid:e81832d5) No inactive management server node found 2025-06-29 04:53:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a267fa03]) (logid:e81832d5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:06,917 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-814bb4ce]) (logid:e037f5d9) HA health check task is running... 2025-06-29 04:53:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3da01680]) (logid:2d0c7ddf) No inactive management server node found 2025-06-29 04:53:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3da01680]) (logid:2d0c7ddf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61253471]) (logid:20f77dc1) No inactive management server node found 2025-06-29 04:53:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-61253471]) (logid:20f77dc1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9469453]) (logid:888968be) No inactive management server node found 2025-06-29 04:53:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9469453]) (logid:888968be) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:10,918 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a87abb46]) (logid:99e3be1c) HA health check task is running... 2025-06-29 04:53:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfe9f86b]) (logid:15b335d7) No inactive management server node found 2025-06-29 04:53:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dfe9f86b]) (logid:15b335d7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c66563c]) (logid:141f7bae) Found 0 routers to update status. 2025-06-29 04:53:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c66563c]) (logid:141f7bae) Found 0 VPC's to update Redundant State. 2025-06-29 04:53:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-4c66563c]) (logid:141f7bae) Found 0 networks to update RvR status. 2025-06-29 04:53:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3614fb5]) (logid:97597da2) No inactive management server node found 2025-06-29 04:53:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3614fb5]) (logid:97597da2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-723ab82f]) (logid:2969d7b6) No inactive management server node found 2025-06-29 04:53:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-723ab82f]) (logid:2969d7b6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:14,919 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-95c4b905]) (logid:60b35483) HA health check task is running... 2025-06-29 04:53:15,412 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8808: Seq 7-8808: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:53:15,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8808 2025-06-29 04:53:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1ce437b]) (logid:b5b788fe) No inactive management server node found 2025-06-29 04:53:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1ce437b]) (logid:b5b788fe) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d56e79c0]) (logid:58819d5a) No inactive management server node found 2025-06-29 04:53:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d56e79c0]) (logid:58819d5a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:18,093 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-9d26342d]) (logid:e0082fc6) AutoScaling Monitor is running... 2025-06-29 04:53:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d2f3c163]) (logid:524c9664) Resetting hosts suitable for reconnect 2025-06-29 04:53:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d2f3c163]) (logid:524c9664) Completed resetting hosts suitable for reconnect 2025-06-29 04:53:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d2f3c163]) (logid:524c9664) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:53:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d2f3c163]) (logid:524c9664) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:53:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d2f3c163]) (logid:524c9664) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:53:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-d2f3c163]) (logid:524c9664) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:53:18,174 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-dc959481]) (logid:c8f1cd2e) Starting to remove all volume_stats rows older than [Sat Jun 28 16:53:18 CEST 2025]. 2025-06-29 04:53:18,175 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-dc959481]) (logid:c8f1cd2e) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:53:18 CEST 2025]. 2025-06-29 04:53:18,920 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-99343864]) (logid:36f95391) HA health check task is running... 2025-06-29 04:53:19,357 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-3a16d9c8]) (logid:7a518351) Starting to remove all vm_stats rows older than [Sat Jun 28 16:53:19 CEST 2025]. 2025-06-29 04:53:19,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-711f3d18]) (logid:6c93f8fa) No inactive management server node found 2025-06-29 04:53:19,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-711f3d18]) (logid:6c93f8fa) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:19,382 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-3a16d9c8]) (logid:7a518351) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:53:19 CEST 2025]. 2025-06-29 04:53:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df6030ea]) (logid:7b0be188) No inactive management server node found 2025-06-29 04:53:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-df6030ea]) (logid:7b0be188) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b83dd58]) (logid:f0206a93) No inactive management server node found 2025-06-29 04:53:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b83dd58]) (logid:f0206a93) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:22,547 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:53:22,550 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:53:22,551 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) Seq 2-4725683383995203900: Routed from 90520738888109 2025-06-29 04:53:22,595 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) Seq 1-4725683383995203900: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:53:22,596 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:53:22,597 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) Seq 2-3926857400090362210: Routed from 90520738888109 2025-06-29 04:53:22,641 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-70442dce]) (logid:5bcaa520) Seq 2-3926857400090362210: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:53:22,921 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-18783ad6]) (logid:31350321) HA health check task is running... 2025-06-29 04:53:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239f8d29]) (logid:a671f91e) No inactive management server node found 2025-06-29 04:53:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239f8d29]) (logid:a671f91e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c115de72]) (logid:8684cc95) No inactive management server node found 2025-06-29 04:53:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c115de72]) (logid:8684cc95) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:25,414 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8809: Seq 7-8809: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:53:25,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8809 2025-06-29 04:53:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0bbd6f2]) (logid:8287de1a) No inactive management server node found 2025-06-29 04:53:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f0bbd6f2]) (logid:8287de1a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:26,922 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-48ac4bc2]) (logid:91ed1fde) HA health check task is running... 2025-06-29 04:53:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-73b4d120]) (logid:60342a5e) Enabled non-edge zones available for scan: 1 2025-06-29 04:53:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-73b4d120]) (logid:60342a5e) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:53:28,092 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-73b4d120]) (logid:60342a5e) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:53:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5d1e3c49]) (logid:0ae51187) Enabled non-edge zones available for scan: 1 2025-06-29 04:53:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-5d1e3c49]) (logid:0ae51187) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:53:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4275a51]) (logid:1ee94f9a) No inactive management server node found 2025-06-29 04:53:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a4275a51]) (logid:1ee94f9a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4ddb5da]) (logid:404b0f6a) No inactive management server node found 2025-06-29 04:53:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e4ddb5da]) (logid:404b0f6a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:29,965 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:53:29,967 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:53:29,968 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) Seq 2-4725683383995203901: Routed from 90520738888109 2025-06-29 04:53:30,047 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) Seq 1-4725683383995203901: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:53:30,082 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:53:30,082 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) Seq 2-3926857400090362211: Routed from 90520738888109 2025-06-29 04:53:30,185 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-3ba51e0a]) (logid:9a5ed969) Seq 2-3926857400090362211: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:53:30,923 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f0924d24]) (logid:58f82c38) HA health check task is running... 2025-06-29 04:53:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-164b9d37]) (logid:c88d914b) No inactive management server node found 2025-06-29 04:53:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-164b9d37]) (logid:c88d914b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:32,061 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) StorageCollector is running... 2025-06-29 04:53:32,065 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:53:32,065 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:53:32,066 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:53:32,066 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-3938116399158788180: Routed from 90520738888109 2025-06-29 04:53:32,142 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 4-3938116399158788180: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:53:32,146 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:53:32,146 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:53:32,146 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:53:32,147 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-4725683383995203902: Routed from 90520738888109 2025-06-29 04:53:32,195 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 1-4725683383995203902: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:53:32,206 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:53:32,206 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:53:32,206 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:53:32,207 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-3926857400090362212: Routed from 90520738888109 2025-06-29 04:53:32,225 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-3926857400090362212: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:53:32,235 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:53:32,235 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:53:32,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:53:32,236 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-4725683383995203903: Routed from 90520738888109 2025-06-29 04:53:32,279 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 1-4725683383995203903: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:53:32,281 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:53:32,281 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:53:32,281 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:53:32,282 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-3926857400090362213: Routed from 90520738888109 2025-06-29 04:53:32,326 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-bf8aeeaf]) (logid:212115ec) Seq 2-3926857400090362213: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:53:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0ea0576]) (logid:e1401e6f) No inactive management server node found 2025-06-29 04:53:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0ea0576]) (logid:e1401e6f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5440ff91]) (logid:79a0814b) No inactive management server node found 2025-06-29 04:53:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5440ff91]) (logid:79a0814b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:34,925 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a0545b6d]) (logid:faec8eb3) HA health check task is running... 2025-06-29 04:53:35,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8810: Seq 7-8810: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:53:35,422 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8810 2025-06-29 04:53:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3645163]) (logid:c62b1aac) No inactive management server node found 2025-06-29 04:53:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3645163]) (logid:c62b1aac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc84d88a]) (logid:41431c1b) No inactive management server node found 2025-06-29 04:53:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fc84d88a]) (logid:41431c1b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1bcff3c]) (logid:8954843d) No inactive management server node found 2025-06-29 04:53:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f1bcff3c]) (logid:8954843d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:38,926 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-42c5b697]) (logid:be83d017) HA health check task is running... 2025-06-29 04:53:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d0dd1d5]) (logid:8a8cfac8) No inactive management server node found 2025-06-29 04:53:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0d0dd1d5]) (logid:8a8cfac8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2214fe0e]) (logid:c27eb8ff) No inactive management server node found 2025-06-29 04:53:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2214fe0e]) (logid:c27eb8ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:42,927 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-95201d81]) (logid:f8bb6981) HA health check task is running... 2025-06-29 04:53:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-83990349]) (logid:9c1df303) DbCollector is running... 2025-06-29 04:53:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b520f083]) (logid:a50cb1a0) Found 0 routers to update status. 2025-06-29 04:53:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b520f083]) (logid:a50cb1a0) Found 0 VPC's to update Redundant State. 2025-06-29 04:53:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-b520f083]) (logid:a50cb1a0) Found 0 networks to update RvR status. 2025-06-29 04:53:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d3e896f]) (logid:d68731f0) No inactive management server node found 2025-06-29 04:53:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7d3e896f]) (logid:d68731f0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) ManagementServerCollector is running... 2025-06-29 04:53:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) DB properties were already loaded 2025-06-29 04:53:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:53:44,057 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Execution of process [4068126] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:53:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Exit value of process [4068126] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:53:44,058 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Process [4068126] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:53:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) usage install: null 2025-06-29 04:53:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Sessions active: 0 2025-06-29 04:53:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:53:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068129] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:53:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:53:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068134] for command [/bin/bash -c uname -r ]. 2025-06-29 04:53:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:53:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068135] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:53:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) system memory from /proc: 4887515136 2025-06-29 04:53:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:53:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068140] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:53:44,082 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) free memory from /proc: 343846912 2025-06-29 04:53:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:53:44,093 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068145] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:53:44,094 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) used memory from /proc: 989936 2025-06-29 04:53:44,094 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:53:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068146] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:53:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:53:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068147] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:53:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:53:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068148] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:53:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:53:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068155] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:53:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:53:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Successfully executed process [4068156] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail buffersdirect.count, 96 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail buffersdirect.used, 328650 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail buffersmapped.capacity, 0 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail buffersmapped.count, 0 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail buffersmapped.used, 0 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail jvmuptime, 4683831 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memoryheap.committed, 671088640 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memoryheap.init, 77594624 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memoryheap.usage, 0.1875024516318598 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorynon-heap.committed, 199098368 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorynon-heap.max, -1 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorynon-heap.usage, -1.95848392E8 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorynon-heap.used, 195848392 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.119168221807518 2025-06-29 04:53:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2216212050521545 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.Compressed-Class-Space.usage, 0.012398816645145416 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.Metaspace.usage, 0.9929780396995079 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.PS-Eden-Space.usage, 0.24244426730398824 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorytotal.committed, 870187008 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorytotal.init, 85262336 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorytotal.max, 1908932607 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail memorytotal.used, 553990752 2025-06-29 04:53:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail threadsdeadlocks, [] 2025-06-29 04:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail threadsnew.count, 0 2025-06-29 04:53:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:53:44,132 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-a7da7da7]) (logid:b01eb57c) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":313871296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4683762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":553065736,"totalCommitted":870187008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357929544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":343846912,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.26,0.2],"systemCyclesUsage":[66424638,53843789,4590407493],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:53:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 155, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":313871296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4683762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":553065736,"totalCommitted":870187008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357929544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":343846912,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.26,0.2],"systemCyclesUsage":[66424638,53843789,4590407493],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:53:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:53:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-33:[ctx-f0b5445b]) (logid:4cd1c361) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":313871296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4683762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":553065736,"totalCommitted":870187008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357929544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":343846912,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.26,0.2],"systemCyclesUsage":[66424638,53843789,4590407493],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:53:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:53:44,138 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:[ctx-7d28bffe]) (logid:ca5a332b) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 155, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:53:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":313871296,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4683762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.12,"totalInit":85262336,"totalUsed":553065736,"totalCommitted":870187008,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":357929544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":343846912,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.12,0.26,0.2],"systemCyclesUsage":[66424638,53843789,4590407493],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:53:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ffafa29]) (logid:96beac4d) No inactive management server node found 2025-06-29 04:53:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ffafa29]) (logid:96beac4d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:45,414 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8811: Seq 7-8811: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:53:45,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8811 2025-06-29 04:53:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:53:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:53:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a39ea9ee]) (logid:87ce392f) No inactive management server node found 2025-06-29 04:53:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a39ea9ee]) (logid:87ce392f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:53:46,643 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:53:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:53:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:53:46,928 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-008091ac]) (logid:706a23d8) HA health check task is running... 2025-06-29 04:53:47,079 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:53:47,080 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:53:47,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:53:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:53:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-046f9d06]) (logid:c3e87a28) No inactive management server node found 2025-06-29 04:53:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-046f9d06]) (logid:c3e87a28) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-46aac2b6]) (logid:af07c9f0) VM scheduler.poll is being called at 2025-06-29T02:54:00+0000 2025-06-29 04:53:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-46aac2b6]) (logid:af07c9f0) Got 0 scheduled jobs to be executed at 2025-06-29T02:54:00+0000 2025-06-29 04:53:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-46aac2b6]) (logid:af07c9f0) Cleaned up 0 VM scheduled job entries 2025-06-29 04:53:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-2be2a7a3]) (logid:ab903e3c) VM Operation Thread Running 2025-06-29 04:53:48,352 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5f9895b0]) (logid:fd16a276) ClusterDRS.poll is being called at 2025-06-29T02:54:00+0000 2025-06-29 04:53:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-5f9895b0]) (logid:fd16a276) Removed 0 old drs migration plans 2025-06-29 04:53:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42d23bf0]) (logid:4f2f06b4) No inactive management server node found 2025-06-29 04:53:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42d23bf0]) (logid:4f2f06b4) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16c6766f]) (logid:6ac3e281) No inactive management server node found 2025-06-29 04:53:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-16c6766f]) (logid:6ac3e281) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:50,929 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-54d4ad96]) (logid:37fb585c) HA health check task is running... 2025-06-29 04:53:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9bb170c]) (logid:04758b6a) No inactive management server node found 2025-06-29 04:53:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b9bb170c]) (logid:04758b6a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7675940b]) (logid:16c257f8) No inactive management server node found 2025-06-29 04:53:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7675940b]) (logid:16c257f8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:54,930 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-978cf7e4]) (logid:3d8f019d) HA health check task is running... 2025-06-29 04:53:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0410bc13]) (logid:bb60ffb8) No inactive management server node found 2025-06-29 04:53:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0410bc13]) (logid:bb60ffb8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:55,414 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8813: Seq 7-8813: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:53:55,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8813 2025-06-29 04:53:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b718dda]) (logid:f950c020) No inactive management server node found 2025-06-29 04:53:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b718dda]) (logid:f950c020) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b90339b3]) (logid:5974d4c5) Enabled non-edge zones available for scan: 1 2025-06-29 04:53:58,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-b90339b3]) (logid:5974d4c5) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:53:58,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-b90339b3]) (logid:5974d4c5) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:53:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e1b6bbd7]) (logid:eea0e689) Enabled non-edge zones available for scan: 1 2025-06-29 04:53:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-e1b6bbd7]) (logid:eea0e689) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:53:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b60b2cc5]) (logid:3f3caf13) No inactive management server node found 2025-06-29 04:53:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b60b2cc5]) (logid:3f3caf13) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:53:58,931 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-511a325d]) (logid:b98ab951) HA health check task is running... 2025-06-29 04:53:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0552a21]) (logid:6af7590b) No inactive management server node found 2025-06-29 04:53:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a0552a21]) (logid:6af7590b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:00,414 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8814: Seq 7-8814: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:00,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8814 2025-06-29 04:54:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-398613f2]) (logid:2c46fe5d) No inactive management server node found 2025-06-29 04:54:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-398613f2]) (logid:2c46fe5d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ab5447c]) (logid:fc139296) No inactive management server node found 2025-06-29 04:54:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ab5447c]) (logid:fc139296) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:02,933 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-b6b11a2f]) (logid:5a724964) HA health check task is running... 2025-06-29 04:54:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b7fb16b]) (logid:69148fd6) No inactive management server node found 2025-06-29 04:54:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b7fb16b]) (logid:69148fd6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:05,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6542dc11]) (logid:39f1a6f1) No inactive management server node found 2025-06-29 04:54:05,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6542dc11]) (logid:39f1a6f1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:06,934 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-c5839be8]) (logid:59be206d) HA health check task is running... 2025-06-29 04:54:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e228b634]) (logid:76c4b68f) No inactive management server node found 2025-06-29 04:54:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e228b634]) (logid:76c4b68f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91584213]) (logid:a49dd550) No inactive management server node found 2025-06-29 04:54:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91584213]) (logid:a49dd550) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:10,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f067bda]) (logid:98368d79) No inactive management server node found 2025-06-29 04:54:10,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f067bda]) (logid:98368d79) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:10,415 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8815: Seq 7-8815: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:10,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8815 2025-06-29 04:54:10,935 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0288e456]) (logid:e026e553) HA health check task is running... 2025-06-29 04:54:11,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64b3a2a5]) (logid:6d4bfdbe) No inactive management server node found 2025-06-29 04:54:11,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64b3a2a5]) (logid:6d4bfdbe) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d6e40b1]) (logid:2a074e8c) Found 0 routers to update status. 2025-06-29 04:54:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d6e40b1]) (logid:2a074e8c) Found 0 VPC's to update Redundant State. 2025-06-29 04:54:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-8d6e40b1]) (logid:2a074e8c) Found 0 networks to update RvR status. 2025-06-29 04:54:13,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb97d326]) (logid:8e82668c) No inactive management server node found 2025-06-29 04:54:13,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cb97d326]) (logid:8e82668c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8395c862]) (logid:0b60d7cf) No inactive management server node found 2025-06-29 04:54:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8395c862]) (logid:0b60d7cf) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:14,936 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-235be62f]) (logid:cdb2cdf3) HA health check task is running... 2025-06-29 04:54:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-708ca779]) (logid:a4cac2a5) No inactive management server node found 2025-06-29 04:54:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-708ca779]) (logid:a4cac2a5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:17,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-395fabfc]) (logid:af1d7412) No inactive management server node found 2025-06-29 04:54:17,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-395fabfc]) (logid:af1d7412) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:18,096 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-3:[ctx-a1e48d64]) (logid:705c4400) AutoScaling Monitor is running... 2025-06-29 04:54:18,177 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-48e7a088]) (logid:5e6f7287) Starting to remove all volume_stats rows older than [Sat Jun 28 16:54:18 CEST 2025]. 2025-06-29 04:54:18,177 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-5:[ctx-48e7a088]) (logid:5e6f7287) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:54:18 CEST 2025]. 2025-06-29 04:54:18,938 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5e8a4db6]) (logid:56531c22) HA health check task is running... 2025-06-29 04:54:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-303bf36f]) (logid:a404eee3) No inactive management server node found 2025-06-29 04:54:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-303bf36f]) (logid:a404eee3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:19,383 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-ba79975f]) (logid:e2735f6f) Starting to remove all vm_stats rows older than [Sat Jun 28 16:54:19 CEST 2025]. 2025-06-29 04:54:19,407 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-4:[ctx-ba79975f]) (logid:e2735f6f) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:54:19 CEST 2025]. 2025-06-29 04:54:20,415 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8816: Seq 7-8816: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:20,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8816 2025-06-29 04:54:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-151b3a93]) (logid:13740694) No inactive management server node found 2025-06-29 04:54:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-151b3a93]) (logid:13740694) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:54:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f362dedd]) (logid:c8951fa6) No inactive management server node found 2025-06-29 04:54:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f362dedd]) (logid:c8951fa6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:22,644 DEBUG [c.c.s.S.HostCollector] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:54:22,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:54:22,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) Seq 2-4725683383995203904: Routed from 90520738888109 2025-06-29 04:54:22,689 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) Seq 1-4725683383995203904: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:54:22,691 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:54:22,691 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) Seq 2-3926857400090362214: Routed from 90520738888109 2025-06-29 04:54:22,735 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-c7f7fc3b]) (logid:e43147d7) Seq 2-3926857400090362214: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:54:22,939 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-626e08d7]) (logid:56690676) HA health check task is running... 2025-06-29 04:54:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f086114]) (logid:eb9bbef7) No inactive management server node found 2025-06-29 04:54:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7f086114]) (logid:eb9bbef7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11b5767d]) (logid:050bd080) No inactive management server node found 2025-06-29 04:54:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-11b5767d]) (logid:050bd080) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-006aa216]) (logid:27e5f2b2) No inactive management server node found 2025-06-29 04:54:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-006aa216]) (logid:27e5f2b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:26,940 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-b592a454]) (logid:2394e850) HA health check task is running... 2025-06-29 04:54:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-db74fc2e]) (logid:974427e3) Enabled non-edge zones available for scan: 1 2025-06-29 04:54:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-db74fc2e]) (logid:974427e3) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:54:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-db74fc2e]) (logid:974427e3) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:54:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f32831d7]) (logid:5515b331) Enabled non-edge zones available for scan: 1 2025-06-29 04:54:28,282 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f32831d7]) (logid:5515b331) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:54:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8265392c]) (logid:0836a9d1) No inactive management server node found 2025-06-29 04:54:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8265392c]) (logid:0836a9d1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0d9eaf6]) (logid:0cc0d795) No inactive management server node found 2025-06-29 04:54:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d0d9eaf6]) (logid:0cc0d795) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:30,240 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:54:30,243 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:54:30,244 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) Seq 2-4725683383995203905: Routed from 90520738888109 2025-06-29 04:54:30,297 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) Seq 1-4725683383995203905: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:54:30,332 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:54:30,333 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) Seq 2-3926857400090362215: Routed from 90520738888109 2025-06-29 04:54:30,416 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8817: Seq 7-8817: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:30,423 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-d77869ea]) (logid:1e03b127) Seq 2-3926857400090362215: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:54:30,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8817 2025-06-29 04:54:30,941 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e052b2f3]) (logid:21bacdff) HA health check task is running... 2025-06-29 04:54:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84ee97dc]) (logid:478005ac) No inactive management server node found 2025-06-29 04:54:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-84ee97dc]) (logid:478005ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:32,327 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) StorageCollector is running... 2025-06-29 04:54:32,334 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:54:32,335 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:54:32,335 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:54:32,336 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 2-3938116399158788181: Routed from 90520738888109 2025-06-29 04:54:32,450 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 4-3938116399158788181: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:54:32,454 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:54:32,454 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:54:32,454 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:54:32,456 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 2-4725683383995203906: Routed from 90520738888109 2025-06-29 04:54:32,503 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 1-4725683383995203906: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:54:32,514 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:54:32,514 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:54:32,514 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:54:32,515 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 2-4725683383995203907: Routed from 90520738888109 2025-06-29 04:54:32,603 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 1-4725683383995203907: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:54:32,615 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:54:32,615 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:54:32,615 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:54:32,616 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 2-4725683383995203908: Routed from 90520738888109 2025-06-29 04:54:32,700 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 1-4725683383995203908: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:54:32,702 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:54:32,702 DEBUG [c.c.h.XenServerGuru] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:54:32,702 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:54:32,703 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 2-3926857400090362216: Routed from 90520738888109 2025-06-29 04:54:32,707 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-fb1c3ceb]) (logid:4921252e) Seq 2-3926857400090362216: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:54:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27e7a2df]) (logid:3b3b69bd) No inactive management server node found 2025-06-29 04:54:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-27e7a2df]) (logid:3b3b69bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:34,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c23e3c3d]) (logid:afc52a02) No inactive management server node found 2025-06-29 04:54:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c23e3c3d]) (logid:afc52a02) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:34,943 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-8abe78f8]) (logid:9cb89c0b) HA health check task is running... 2025-06-29 04:54:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51304493]) (logid:5aa820e2) No inactive management server node found 2025-06-29 04:54:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51304493]) (logid:5aa820e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0aa6dab3]) (logid:38369427) No inactive management server node found 2025-06-29 04:54:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0aa6dab3]) (logid:38369427) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75b9206e]) (logid:ebf74f7d) No inactive management server node found 2025-06-29 04:54:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-75b9206e]) (logid:ebf74f7d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:38,944 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-87bcc075]) (logid:da405c3f) HA health check task is running... 2025-06-29 04:54:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f00d781]) (logid:d9c985e3) No inactive management server node found 2025-06-29 04:54:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f00d781]) (logid:d9c985e3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:40,416 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8818: Seq 7-8818: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:40,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8818 2025-06-29 04:54:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d08567]) (logid:143366bd) No inactive management server node found 2025-06-29 04:54:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a7d08567]) (logid:143366bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:42,945 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-11922d80]) (logid:9255411d) HA health check task is running... 2025-06-29 04:54:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-3:[ctx-dce32acc]) (logid:ac92e279) DbCollector is running... 2025-06-29 04:54:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ba6702b0]) (logid:afc968e3) Found 0 routers to update status. 2025-06-29 04:54:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ba6702b0]) (logid:afc968e3) Found 0 VPC's to update Redundant State. 2025-06-29 04:54:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-ba6702b0]) (logid:afc968e3) Found 0 networks to update RvR status. 2025-06-29 04:54:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e5c4996]) (logid:d971708b) No inactive management server node found 2025-06-29 04:54:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1e5c4996]) (logid:d971708b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) ManagementServerCollector is running... 2025-06-29 04:54:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) DB properties were already loaded 2025-06-29 04:54:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:54:44,057 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Execution of process [4068164] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:54:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Exit value of process [4068164] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:54:44,058 WARN [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Process [4068164] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:54:44,058 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) usage install: null 2025-06-29 04:54:44,058 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Sessions active: 0 2025-06-29 04:54:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:54:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068167] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:54:44,066 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:54:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068172] for command [/bin/bash -c uname -r ]. 2025-06-29 04:54:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:54:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068173] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:54:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) system memory from /proc: 4887515136 2025-06-29 04:54:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:54:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068178] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:54:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) free memory from /proc: 350318592 2025-06-29 04:54:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:54:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068183] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:54:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) used memory from /proc: 989936 2025-06-29 04:54:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:54:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068184] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:54:44,101 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:54:44,112 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068185] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:54:44,113 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:54:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068186] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:54:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:54:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068193] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:54:44,123 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:54:44,127 DEBUG [c.c.u.s.Script] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Successfully executed process [4068194] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail buffersdirect.count, 96 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail buffersdirect.used, 328650 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail buffersmapped.capacity, 0 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail buffersmapped.count, 0 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail buffersmapped.used, 0 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:54:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail jvmuptime, 4743833 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memoryheap.committed, 671088640 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memoryheap.init, 77594624 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memoryheap.usage, 0.1999138567808466 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorynon-heap.committed, 199163904 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorynon-heap.max, -1 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorynon-heap.usage, -1.95927672E8 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorynon-heap.used, 195927672 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.11968266795521194 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22173055453727464 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.Compressed-Class-Space.usage, 0.012398816645145416 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.Metaspace.usage, 0.9929966877416238 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.PS-Eden-Space.usage, 0.2792139387014341 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorytotal.committed, 870252544 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorytotal.init, 85262336 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorytotal.max, 1908932607 2025-06-29 04:54:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail memorytotal.used, 577732232 2025-06-29 04:54:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail threadsdeadlocks, [] 2025-06-29 04:54:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail threadsnew.count, 0 2025-06-29 04:54:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:54:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:[ctx-1f6ab9ae]) (logid:b199dc29) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":290046864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4743761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":576969448,"totalCommitted":870252544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381622080,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":350318592,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.29,0.21],"systemCyclesUsage":[66424975,53844085,4590430618],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:54:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 157, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":290046864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4743761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":576969448,"totalCommitted":870252544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381622080,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":350318592,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.29,0.21],"systemCyclesUsage":[66424975,53844085,4590430618],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:54:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:54:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:54:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 157, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":290046864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4743761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":576969448,"totalCommitted":870252544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381622080,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":350318592,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.29,0.21],"systemCyclesUsage":[66424975,53844085,4590430618],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:54:44,140 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-34:[ctx-ebae0549]) (logid:4d553f07) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:54:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":290046864,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4743761,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.41,"totalInit":85262336,"totalUsed":576969448,"totalCommitted":870252544,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":381622080,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":350318592,"systemMemoryUsed":989936,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.41,0.29,0.21],"systemCyclesUsage":[66424975,53844085,4590430618],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:54:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57d054df]) (logid:276c2960) No inactive management server node found 2025-06-29 04:54:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57d054df]) (logid:276c2960) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:45,416 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8819: Seq 7-8819: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:45,426 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8819 2025-06-29 04:54:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:54:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:54:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f25894d2]) (logid:c1162cd8) No inactive management server node found 2025-06-29 04:54:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f25894d2]) (logid:c1162cd8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:46,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:54:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:54:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:54:46,650 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:54:46,946 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-bea24c24]) (logid:ecd87ca9) HA health check task is running... 2025-06-29 04:54:47,080 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:54:47,082 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:54:47,083 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:54:47,085 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:54:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c833c56]) (logid:0c09c6e2) No inactive management server node found 2025-06-29 04:54:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c833c56]) (logid:0c09c6e2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:48,095 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-112bebae]) (logid:e2b106d3) Resetting hosts suitable for reconnect 2025-06-29 04:54:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-112bebae]) (logid:e2b106d3) Completed resetting hosts suitable for reconnect 2025-06-29 04:54:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-112bebae]) (logid:e2b106d3) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:54:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-112bebae]) (logid:e2b106d3) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:54:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-112bebae]) (logid:e2b106d3) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:54:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-112bebae]) (logid:e2b106d3) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:54:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-107c83d0]) (logid:abe60893) VM scheduler.poll is being called at 2025-06-29T02:55:00+0000 2025-06-29 04:54:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-107c83d0]) (logid:abe60893) Got 0 scheduled jobs to be executed at 2025-06-29T02:55:00+0000 2025-06-29 04:54:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-107c83d0]) (logid:abe60893) Cleaned up 0 VM scheduled job entries 2025-06-29 04:54:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-9c852cb2]) (logid:b4c61ee9) VM Operation Thread Running 2025-06-29 04:54:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6a14fb12]) (logid:a6d73f95) ClusterDRS.poll is being called at 2025-06-29T02:55:00+0000 2025-06-29 04:54:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-6a14fb12]) (logid:a6d73f95) Removed 0 old drs migration plans 2025-06-29 04:54:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b16217d9]) (logid:75789a90) No inactive management server node found 2025-06-29 04:54:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b16217d9]) (logid:75789a90) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3c8f6a7]) (logid:118bf38e) No inactive management server node found 2025-06-29 04:54:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c3c8f6a7]) (logid:118bf38e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:50,947 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-105b6ada]) (logid:69ee3284) HA health check task is running... 2025-06-29 04:54:52,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a25bf1]) (logid:9e274471) No inactive management server node found 2025-06-29 04:54:52,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d4a25bf1]) (logid:9e274471) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e565288]) (logid:f6b3c544) No inactive management server node found 2025-06-29 04:54:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e565288]) (logid:f6b3c544) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:54,948 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-275f777d]) (logid:e71209a5) HA health check task is running... 2025-06-29 04:54:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f2b696c]) (logid:a2473ac5) No inactive management server node found 2025-06-29 04:54:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5f2b696c]) (logid:a2473ac5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:55,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8821: Seq 7-8821: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:54:55,427 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8821 2025-06-29 04:54:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a42fc89]) (logid:e45046a6) No inactive management server node found 2025-06-29 04:54:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a42fc89]) (logid:e45046a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-972d91f3]) (logid:581f3523) Enabled non-edge zones available for scan: 1 2025-06-29 04:54:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-972d91f3]) (logid:581f3523) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:54:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-972d91f3]) (logid:581f3523) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:54:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-40603102]) (logid:954d18f5) Enabled non-edge zones available for scan: 1 2025-06-29 04:54:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-40603102]) (logid:954d18f5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:54:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68507031]) (logid:99e24fc5) No inactive management server node found 2025-06-29 04:54:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68507031]) (logid:99e24fc5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:54:58,949 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-4f658ce2]) (logid:febc01af) HA health check task is running... 2025-06-29 04:54:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76650597]) (logid:74c38f88) No inactive management server node found 2025-06-29 04:54:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76650597]) (logid:74c38f88) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:01,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7679861d]) (logid:0cc6c67a) No inactive management server node found 2025-06-29 04:55:01,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7679861d]) (logid:0cc6c67a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-207fe107]) (logid:95c159a6) No inactive management server node found 2025-06-29 04:55:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-207fe107]) (logid:95c159a6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:02,950 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-0751e0c8]) (logid:f74fd2db) HA health check task is running... 2025-06-29 04:55:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-579048fc]) (logid:392624d4) No inactive management server node found 2025-06-29 04:55:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-579048fc]) (logid:392624d4) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:05,417 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8822: Seq 7-8822: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:55:05,426 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8822 2025-06-29 04:55:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eada6e63]) (logid:d0b63bac) No inactive management server node found 2025-06-29 04:55:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eada6e63]) (logid:d0b63bac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:06,951 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-0255d893]) (logid:337cc82f) HA health check task is running... 2025-06-29 04:55:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-930bd236]) (logid:588ffd54) No inactive management server node found 2025-06-29 04:55:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-930bd236]) (logid:588ffd54) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc162d7]) (logid:3b8a6812) No inactive management server node found 2025-06-29 04:55:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bfc162d7]) (logid:3b8a6812) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-097b2bd3]) (logid:1070ea10) No inactive management server node found 2025-06-29 04:55:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-097b2bd3]) (logid:1070ea10) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:10,953 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-b3b766a3]) (logid:326276a1) HA health check task is running... 2025-06-29 04:55:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ed513ae]) (logid:c0eabdbc) No inactive management server node found 2025-06-29 04:55:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5ed513ae]) (logid:c0eabdbc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-663517f8]) (logid:a310b0a3) Found 0 routers to update status. 2025-06-29 04:55:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-663517f8]) (logid:a310b0a3) Found 0 VPC's to update Redundant State. 2025-06-29 04:55:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-663517f8]) (logid:a310b0a3) Found 0 networks to update RvR status. 2025-06-29 04:55:13,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-740e6b3b]) (logid:f5e66a76) No inactive management server node found 2025-06-29 04:55:13,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-740e6b3b]) (logid:f5e66a76) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b28dea3]) (logid:587ba936) No inactive management server node found 2025-06-29 04:55:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b28dea3]) (logid:587ba936) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:14,954 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-174d5efb]) (logid:ad3ae745) HA health check task is running... 2025-06-29 04:55:15,418 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8823: Seq 7-8823: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:55:15,427 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8823 2025-06-29 04:55:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d66311a]) (logid:928774f0) No inactive management server node found 2025-06-29 04:55:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2d66311a]) (logid:928774f0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b147f6e]) (logid:584e1ef0) No inactive management server node found 2025-06-29 04:55:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6b147f6e]) (logid:584e1ef0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:18,098 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-4:[ctx-bd6d8572]) (logid:1a016cb6) AutoScaling Monitor is running... 2025-06-29 04:55:18,180 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-8fb4f3f7]) (logid:299cf895) Starting to remove all volume_stats rows older than [Sat Jun 28 16:55:18 CEST 2025]. 2025-06-29 04:55:18,180 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-1:[ctx-8fb4f3f7]) (logid:299cf895) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:55:18 CEST 2025]. 2025-06-29 04:55:18,955 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-19290238]) (logid:83bb718e) HA health check task is running... 2025-06-29 04:55:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cccc2ec]) (logid:32e38a82) No inactive management server node found 2025-06-29 04:55:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2cccc2ec]) (logid:32e38a82) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:19,408 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6704f01e]) (logid:a9b19671) Starting to remove all vm_stats rows older than [Sat Jun 28 16:55:19 CEST 2025]. 2025-06-29 04:55:19,432 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-6:[ctx-6704f01e]) (logid:a9b19671) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:55:19 CEST 2025]. 2025-06-29 04:55:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf46ecd3]) (logid:52fe9c38) No inactive management server node found 2025-06-29 04:55:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bf46ecd3]) (logid:52fe9c38) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d261ad05]) (logid:a79ef963) No inactive management server node found 2025-06-29 04:55:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d261ad05]) (logid:a79ef963) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:55:22,738 DEBUG [c.c.s.S.HostCollector] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:55:22,740 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:55:22,741 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) Seq 2-4725683383995203909: Routed from 90520738888109 2025-06-29 04:55:22,784 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) Seq 1-4725683383995203909: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:55:22,786 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:55:22,787 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) Seq 2-3926857400090362217: Routed from 90520738888109 2025-06-29 04:55:22,831 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-773ec1cf]) (logid:3a9160ac) Seq 2-3926857400090362217: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:55:22,956 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-d042c0fe]) (logid:bb85093d) HA health check task is running... 2025-06-29 04:55:23,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57e4c6c3]) (logid:0d33539b) No inactive management server node found 2025-06-29 04:55:23,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57e4c6c3]) (logid:0d33539b) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-054e09f1]) (logid:8ebe8acc) No inactive management server node found 2025-06-29 04:55:25,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-054e09f1]) (logid:8ebe8acc) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:25,419 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8824: Seq 7-8824: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:55:25,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8824 2025-06-29 04:55:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e077106]) (logid:e2230858) No inactive management server node found 2025-06-29 04:55:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e077106]) (logid:e2230858) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:26,957 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-1ae11ea4]) (logid:576acc51) HA health check task is running... 2025-06-29 04:55:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-534df3b3]) (logid:1a218a97) Enabled non-edge zones available for scan: 1 2025-06-29 04:55:28,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-534df3b3]) (logid:1a218a97) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:55:28,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-534df3b3]) (logid:1a218a97) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:55:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7dae7b46]) (logid:eab8ae1b) Enabled non-edge zones available for scan: 1 2025-06-29 04:55:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7dae7b46]) (logid:eab8ae1b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:55:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5089a301]) (logid:ac4be630) No inactive management server node found 2025-06-29 04:55:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5089a301]) (logid:ac4be630) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09682f33]) (logid:9961a2a6) No inactive management server node found 2025-06-29 04:55:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-09682f33]) (logid:9961a2a6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:30,479 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:55:30,482 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:55:30,483 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) Seq 2-4725683383995203910: Routed from 90520738888109 2025-06-29 04:55:30,537 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) Seq 1-4725683383995203910: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:55:30,573 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:55:30,574 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) Seq 2-3926857400090362218: Routed from 90520738888109 2025-06-29 04:55:30,665 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-047be612]) (logid:4a12ea37) Seq 2-3926857400090362218: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:55:30,958 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8cc3ef35]) (logid:baa49531) HA health check task is running... 2025-06-29 04:55:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a846b947]) (logid:e1dee777) No inactive management server node found 2025-06-29 04:55:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a846b947]) (logid:e1dee777) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:32,707 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) StorageCollector is running... 2025-06-29 04:55:32,714 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:55:32,714 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:55:32,714 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:55:32,715 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3938116399158788182: Routed from 90520738888109 2025-06-29 04:55:32,771 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 4-3938116399158788182: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:55:32,776 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:55:32,776 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:55:32,776 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:55:32,778 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3926857400090362219: Routed from 90520738888109 2025-06-29 04:55:32,789 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3926857400090362219: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:55:32,800 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:55:32,800 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:55:32,800 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:55:32,801 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3926857400090362220: Routed from 90520738888109 2025-06-29 04:55:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdd40e25]) (logid:1c4c1b7b) No inactive management server node found 2025-06-29 04:55:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cdd40e25]) (logid:1c4c1b7b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:32,894 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3926857400090362220: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:55:32,905 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:55:32,906 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:55:32,906 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:55:32,906 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-4725683383995203911: Routed from 90520738888109 2025-06-29 04:55:32,950 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 1-4725683383995203911: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:55:32,952 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:55:32,952 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:55:32,952 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:55:32,953 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3926857400090362221: Routed from 90520738888109 2025-06-29 04:55:32,998 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-59233838]) (logid:d455eef4) Seq 2-3926857400090362221: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:55:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67baf6d6]) (logid:813a7d1d) No inactive management server node found 2025-06-29 04:55:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-67baf6d6]) (logid:813a7d1d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:34,959 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-5984e7c2]) (logid:cd147e46) HA health check task is running... 2025-06-29 04:55:35,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8825: Seq 7-8825: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:55:35,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8825 2025-06-29 04:55:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-979432ef]) (logid:eea2ad90) No inactive management server node found 2025-06-29 04:55:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-979432ef]) (logid:eea2ad90) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bfe418d]) (logid:9c1744e6) No inactive management server node found 2025-06-29 04:55:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0bfe418d]) (logid:9c1744e6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7a7a9e9]) (logid:74b5a116) No inactive management server node found 2025-06-29 04:55:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b7a7a9e9]) (logid:74b5a116) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:38,961 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-3449afea]) (logid:f6bbe5bd) HA health check task is running... 2025-06-29 04:55:40,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-501530e2]) (logid:1eb1b125) No inactive management server node found 2025-06-29 04:55:40,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-501530e2]) (logid:1eb1b125) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5d61aa3]) (logid:6e823755) No inactive management server node found 2025-06-29 04:55:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a5d61aa3]) (logid:6e823755) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:42,962 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-fdcf61a3]) (logid:ff2a585d) HA health check task is running... 2025-06-29 04:55:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-92a3b49c]) (logid:7aff21a5) DbCollector is running... 2025-06-29 04:55:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-137cbf2f]) (logid:21c5b4c8) Found 0 routers to update status. 2025-06-29 04:55:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-137cbf2f]) (logid:21c5b4c8) Found 0 VPC's to update Redundant State. 2025-06-29 04:55:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-137cbf2f]) (logid:21c5b4c8) Found 0 networks to update RvR status. 2025-06-29 04:55:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbf45f5c]) (logid:e660944f) No inactive management server node found 2025-06-29 04:55:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bbf45f5c]) (logid:e660944f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) ManagementServerCollector is running... 2025-06-29 04:55:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) DB properties were already loaded 2025-06-29 04:55:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:55:44,060 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Execution of process [4068206] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:55:44,060 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Exit value of process [4068206] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:55:44,060 WARN [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Process [4068206] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:55:44,060 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) usage install: null 2025-06-29 04:55:44,060 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Sessions active: 0 2025-06-29 04:55:44,063 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:55:44,069 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068209] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:55:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:55:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068214] for command [/bin/bash -c uname -r ]. 2025-06-29 04:55:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:55:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068215] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:55:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) system memory from /proc: 4887515136 2025-06-29 04:55:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:55:44,085 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068220] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:55:44,085 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) free memory from /proc: 347766784 2025-06-29 04:55:44,085 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:55:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068225] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:55:44,098 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) used memory from /proc: 990192 2025-06-29 04:55:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:55:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068226] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:55:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:55:44,114 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068227] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:55:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:55:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068228] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:55:44,121 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:55:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068235] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:55:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:55:44,130 DEBUG [c.c.u.s.Script] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Successfully executed process [4068236] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail buffersdirect.capacity, 328650 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail buffersdirect.count, 96 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail buffersdirect.used, 328650 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail buffersmapped.capacity, 0 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail buffersmapped.count, 0 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail buffersmapped.used, 0 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail jvmuptime, 4803836 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memoryheap.committed, 671088640 2025-06-29 04:55:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memoryheap.init, 77594624 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memoryheap.usage, 0.21106394553243443 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorynon-heap.committed, 199229440 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorynon-heap.max, -1 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorynon-heap.usage, -1.96056072E8 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorynon-heap.used, 196056072 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.12037415022660623 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2220065318092445 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.Compressed-Class-Space.usage, 0.012398816645145416 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.Metaspace.usage, 0.9930645300052718 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.PS-Eden-Space.usage, 0.3122468704707842 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorytotal.committed, 870318080 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorytotal.init, 85262336 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorytotal.max, 1908932607 2025-06-29 04:55:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail memorytotal.used, 598962920 2025-06-29 04:55:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail threadsdeadlocks, [] 2025-06-29 04:55:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail threadsnew.count, 0 2025-06-29 04:55:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:55:44,136 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-1:[ctx-310fc9c4]) (logid:426e1dca) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":268719888,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4803764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.26,"totalInit":85262336,"totalUsed":598421512,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402906848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":347766784,"systemMemoryUsed":990192,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.26,0.28,0.21],"systemCyclesUsage":[66425326,53844392,4590453716],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:55:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 159, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":268719888,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4803764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.26,"totalInit":85262336,"totalUsed":598421512,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402906848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":347766784,"systemMemoryUsed":990192,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.26,0.28,0.21],"systemCyclesUsage":[66425326,53844392,4590453716],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:55:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:55:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:55:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 159, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":268719888,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4803764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.26,"totalInit":85262336,"totalUsed":598421512,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402906848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":347766784,"systemMemoryUsed":990192,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.26,0.28,0.21],"systemCyclesUsage":[66425326,53844392,4590453716],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:55:44,144 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-35:[ctx-acbefb20]) (logid:9db67ab7) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:55:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":268719888,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4803764,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.26,"totalInit":85262336,"totalUsed":598421512,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":402906848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":203,"threadsWaitingCount":110,"systemMemoryTotal":4887515136,"systemMemoryFree":347766784,"systemMemoryUsed":990192,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.26,0.28,0.21],"systemCyclesUsage":[66425326,53844392,4590453716],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:55:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3ad4c25]) (logid:c7efbfd5) No inactive management server node found 2025-06-29 04:55:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b3ad4c25]) (logid:c7efbfd5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:45,419 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8826: Seq 7-8826: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:55:45,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8826 2025-06-29 04:55:45,763 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:55:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:55:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c203ae4c]) (logid:9199bcd5) No inactive management server node found 2025-06-29 04:55:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c203ae4c]) (logid:9199bcd5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:55:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:55:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:55:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:55:46,963 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b852cb13]) (logid:c454587e) HA health check task is running... 2025-06-29 04:55:47,134 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:55:47,136 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:55:47,137 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:55:47,139 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:55:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3c76dbe]) (logid:16935c2d) No inactive management server node found 2025-06-29 04:55:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e3c76dbe]) (logid:16935c2d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-40ce2eab]) (logid:1cfe655d) VM scheduler.poll is being called at 2025-06-29T02:56:00+0000 2025-06-29 04:55:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-40ce2eab]) (logid:1cfe655d) Got 0 scheduled jobs to be executed at 2025-06-29T02:56:00+0000 2025-06-29 04:55:48,162 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-40ce2eab]) (logid:1cfe655d) Cleaned up 0 VM scheduled job entries 2025-06-29 04:55:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-3ac6dfa8]) (logid:485a29dd) VM Operation Thread Running 2025-06-29 04:55:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ea6ceea8]) (logid:8281c6ba) ClusterDRS.poll is being called at 2025-06-29T02:56:00+0000 2025-06-29 04:55:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-ea6ceea8]) (logid:8281c6ba) Removed 0 old drs migration plans 2025-06-29 04:55:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad457a20]) (logid:4d740667) No inactive management server node found 2025-06-29 04:55:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad457a20]) (logid:4d740667) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eeaf3a01]) (logid:59ec0c74) No inactive management server node found 2025-06-29 04:55:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eeaf3a01]) (logid:59ec0c74) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:50,964 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-efd56cee]) (logid:4ea8c6f2) HA health check task is running... 2025-06-29 04:55:52,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bc25160]) (logid:6c25aeb9) No inactive management server node found 2025-06-29 04:55:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7bc25160]) (logid:6c25aeb9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51d9f7d7]) (logid:cab93569) No inactive management server node found 2025-06-29 04:55:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51d9f7d7]) (logid:cab93569) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:54,965 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-13f99aeb]) (logid:553ffff6) HA health check task is running... 2025-06-29 04:55:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eebb61c2]) (logid:c1465bcd) No inactive management server node found 2025-06-29 04:55:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eebb61c2]) (logid:c1465bcd) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:55,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8828: Seq 7-8828: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:55:55,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8828 2025-06-29 04:55:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7dfe054]) (logid:cb83128f) No inactive management server node found 2025-06-29 04:55:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f7dfe054]) (logid:cb83128f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5b5bca27]) (logid:3c84ef1a) Enabled non-edge zones available for scan: 1 2025-06-29 04:55:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-5b5bca27]) (logid:3c84ef1a) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:55:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-5b5bca27]) (logid:3c84ef1a) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:55:58,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-32f866fe]) (logid:91fb5dad) Enabled non-edge zones available for scan: 1 2025-06-29 04:55:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-32f866fe]) (logid:91fb5dad) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:55:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-655fd920]) (logid:4834bc03) No inactive management server node found 2025-06-29 04:55:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-655fd920]) (logid:4834bc03) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:55:58,967 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2ee1e41b]) (logid:08ad7db0) HA health check task is running... 2025-06-29 04:55:59,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e89d553]) (logid:754f606a) No inactive management server node found 2025-06-29 04:55:59,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e89d553]) (logid:754f606a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d77a255]) (logid:c00fc8f9) No inactive management server node found 2025-06-29 04:56:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d77a255]) (logid:c00fc8f9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ece05bda]) (logid:53c8e876) No inactive management server node found 2025-06-29 04:56:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ece05bda]) (logid:53c8e876) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:02,968 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-20130ba9]) (logid:d6f7bd53) HA health check task is running... 2025-06-29 04:56:03,288 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 2-3926857400090361857: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a5f2530]) (logid:5603e1cd) No inactive management server node found 2025-06-29 04:56:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0a5f2530]) (logid:5603e1cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:05,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8829: Seq 7-8829: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:05,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8829 2025-06-29 04:56:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1310ebcf]) (logid:309a5ff8) No inactive management server node found 2025-06-29 04:56:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1310ebcf]) (logid:309a5ff8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:06,969 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ba603b6d]) (logid:130feee9) HA health check task is running... 2025-06-29 04:56:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c242f69]) (logid:6930f1ee) No inactive management server node found 2025-06-29 04:56:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c242f69]) (logid:6930f1ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cee32c2e]) (logid:8e4568bb) No inactive management server node found 2025-06-29 04:56:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cee32c2e]) (logid:8e4568bb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83e94268]) (logid:8e98de1f) No inactive management server node found 2025-06-29 04:56:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-83e94268]) (logid:8e98de1f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:10,420 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8830: Seq 7-8830: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:10,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8830 2025-06-29 04:56:10,970 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-b1fdaee0]) (logid:25245d1f) HA health check task is running... 2025-06-29 04:56:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3449e01c]) (logid:edb08dbc) No inactive management server node found 2025-06-29 04:56:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3449e01c]) (logid:edb08dbc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-10079c89]) (logid:8f744129) Found 0 routers to update status. 2025-06-29 04:56:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-10079c89]) (logid:8f744129) Found 0 VPC's to update Redundant State. 2025-06-29 04:56:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-10079c89]) (logid:8f744129) Found 0 networks to update RvR status. 2025-06-29 04:56:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9f57e2d]) (logid:07a15b9a) No inactive management server node found 2025-06-29 04:56:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9f57e2d]) (logid:07a15b9a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f74ba8cc]) (logid:da753791) No inactive management server node found 2025-06-29 04:56:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f74ba8cc]) (logid:da753791) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:14,971 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-9bd575a7]) (logid:fc4c65e8) HA health check task is running... 2025-06-29 04:56:16,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a54131ad]) (logid:b3bb0a91) No inactive management server node found 2025-06-29 04:56:16,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a54131ad]) (logid:b3bb0a91) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:17,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64ea5998]) (logid:1fb295f8) No inactive management server node found 2025-06-29 04:56:17,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-64ea5998]) (logid:1fb295f8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e69735c4]) (logid:6ec4b4f8) Resetting hosts suitable for reconnect 2025-06-29 04:56:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e69735c4]) (logid:6ec4b4f8) Completed resetting hosts suitable for reconnect 2025-06-29 04:56:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e69735c4]) (logid:6ec4b4f8) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:56:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e69735c4]) (logid:6ec4b4f8) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:56:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e69735c4]) (logid:6ec4b4f8) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:56:18,099 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-e69735c4]) (logid:6ec4b4f8) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:56:18,100 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-6:[ctx-4fd72674]) (logid:b01b0330) AutoScaling Monitor is running... 2025-06-29 04:56:18,183 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-ae27f43c]) (logid:f993fdd3) Starting to remove all volume_stats rows older than [Sat Jun 28 16:56:18 CEST 2025]. 2025-06-29 04:56:18,183 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:[ctx-ae27f43c]) (logid:f993fdd3) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:56:18 CEST 2025]. 2025-06-29 04:56:18,972 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-8e8236be]) (logid:bf157f0b) HA health check task is running... 2025-06-29 04:56:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-776d65cb]) (logid:6fb7961d) No inactive management server node found 2025-06-29 04:56:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-776d65cb]) (logid:6fb7961d) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:19,433 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-e746179d]) (logid:3d74961e) Starting to remove all vm_stats rows older than [Sat Jun 28 16:56:19 CEST 2025]. 2025-06-29 04:56:19,456 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-3:[ctx-e746179d]) (logid:3d74961e) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:56:19 CEST 2025]. 2025-06-29 04:56:20,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8831: Seq 7-8831: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:20,431 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8831 2025-06-29 04:56:20,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-647be2cc]) (logid:50561977) No inactive management server node found 2025-06-29 04:56:20,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-647be2cc]) (logid:50561977) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:22,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-817456bf]) (logid:a1a62414) No inactive management server node found 2025-06-29 04:56:22,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-817456bf]) (logid:a1a62414) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:22,834 DEBUG [c.c.s.S.HostCollector] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:56:22,835 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:56:22,837 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) Seq 2-4725683383995203912: Routed from 90520738888109 2025-06-29 04:56:22,880 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) Seq 1-4725683383995203912: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:56:22,880 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:56:22,881 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) Seq 2-3926857400090362222: Routed from 90520738888109 2025-06-29 04:56:22,884 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-3a2f30c9]) (logid:af01a4ec) Seq 2-3926857400090362222: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:56:22,973 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-12a51fba]) (logid:9c2a747b) HA health check task is running... 2025-06-29 04:56:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b73fcc38]) (logid:95c85521) No inactive management server node found 2025-06-29 04:56:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b73fcc38]) (logid:95c85521) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f170ad0d]) (logid:b5ef6888) No inactive management server node found 2025-06-29 04:56:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f170ad0d]) (logid:b5ef6888) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ed72403]) (logid:4159166b) No inactive management server node found 2025-06-29 04:56:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7ed72403]) (logid:4159166b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:26,974 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-ae9aecf4]) (logid:53ca1b05) HA health check task is running... 2025-06-29 04:56:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9094ea9a]) (logid:ce86a352) Enabled non-edge zones available for scan: 1 2025-06-29 04:56:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-9094ea9a]) (logid:ce86a352) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:56:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-9094ea9a]) (logid:ce86a352) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:56:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b59aebef]) (logid:95aba921) Enabled non-edge zones available for scan: 1 2025-06-29 04:56:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-b59aebef]) (logid:95aba921) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:56:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f784dca]) (logid:79cbb1eb) No inactive management server node found 2025-06-29 04:56:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0f784dca]) (logid:79cbb1eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6df9e231]) (logid:6b885117) No inactive management server node found 2025-06-29 04:56:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6df9e231]) (logid:6b885117) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:30,421 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8832: Seq 7-8832: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:30,431 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8832 2025-06-29 04:56:30,719 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:56:30,722 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:56:30,723 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) Seq 2-4725683383995203913: Routed from 90520738888109 2025-06-29 04:56:30,775 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) Seq 1-4725683383995203913: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:56:30,809 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:56:30,810 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) Seq 2-3926857400090362223: Routed from 90520738888109 2025-06-29 04:56:30,898 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-91a91ff8]) (logid:1aff2fb1) Seq 2-3926857400090362223: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:56:30,975 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-780a7c9f]) (logid:eeca02e2) HA health check task is running... 2025-06-29 04:56:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5bede4f]) (logid:27654336) No inactive management server node found 2025-06-29 04:56:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5bede4f]) (logid:27654336) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-883fc87c]) (logid:8dc69b58) No inactive management server node found 2025-06-29 04:56:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-883fc87c]) (logid:8dc69b58) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:32,998 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) StorageCollector is running... 2025-06-29 04:56:33,006 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:56:33,006 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:56:33,006 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:56:33,007 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3938116399158788183: Routed from 90520738888109 2025-06-29 04:56:33,082 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 4-3938116399158788183: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:56:33,085 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:56:33,085 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:56:33,085 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:56:33,086 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3926857400090362224: Routed from 90520738888109 2025-06-29 04:56:33,097 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3926857400090362224: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:56:33,108 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:56:33,108 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:56:33,108 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:56:33,109 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3926857400090362225: Routed from 90520738888109 2025-06-29 04:56:33,211 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3926857400090362225: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:56:33,222 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:56:33,222 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:56:33,222 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:56:33,223 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-4725683383995203914: Routed from 90520738888109 2025-06-29 04:56:33,267 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 1-4725683383995203914: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:56:33,269 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:56:33,269 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:56:33,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:56:33,271 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3926857400090362226: Routed from 90520738888109 2025-06-29 04:56:33,315 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-2b9592b1]) (logid:e89e763b) Seq 2-3926857400090362226: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:56:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9056cd0a]) (logid:c8c74752) No inactive management server node found 2025-06-29 04:56:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9056cd0a]) (logid:c8c74752) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:34,976 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-07058a77]) (logid:ca2e9bf3) HA health check task is running... 2025-06-29 04:56:35,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8833: Seq 7-8833: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:35,434 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8833 2025-06-29 04:56:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-493fff08]) (logid:4a79d36c) No inactive management server node found 2025-06-29 04:56:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-493fff08]) (logid:4a79d36c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6eeea3bd]) (logid:944c2e52) No inactive management server node found 2025-06-29 04:56:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6eeea3bd]) (logid:944c2e52) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-512c52f7]) (logid:a8d07593) No inactive management server node found 2025-06-29 04:56:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-512c52f7]) (logid:a8d07593) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:38,977 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-75338c53]) (logid:6dd35dfe) HA health check task is running... 2025-06-29 04:56:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-079ea0b3]) (logid:002cb351) No inactive management server node found 2025-06-29 04:56:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-079ea0b3]) (logid:002cb351) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:41,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3421c238]) (logid:299377be) No inactive management server node found 2025-06-29 04:56:41,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3421c238]) (logid:299377be) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:42,978 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c8852389]) (logid:40f88cea) HA health check task is running... 2025-06-29 04:56:43,034 INFO [o.a.c.e.o.N.NetworkGarbageCollector] (Network-Scavenger-1:[ctx-3f2962e0]) (logid:425978f5) NetworkGarbageCollector uses '600' seconds for GC interval. 2025-06-29 04:56:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-6:[ctx-0fd1f989]) (logid:d54698bc) DbCollector is running... 2025-06-29 04:56:43,097 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-dd73381d]) (logid:5770d8dd) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 04:56:43,098 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-80ffb836]) (logid:43de9bde) Found 0 running routers. 2025-06-29 04:56:43,098 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-dd73381d]) (logid:5770d8dd) Snapshot scheduler is being called at [2025-06-29T02:56:43+0000]. 2025-06-29 04:56:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7971af8e]) (logid:d9989750) Found 0 routers to update status. 2025-06-29 04:56:43,099 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-dd73381d]) (logid:5770d8dd) There are [0] scheduled snapshots to be executed at [2025-06-29T02:56:43+0000]. 2025-06-29 04:56:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7971af8e]) (logid:d9989750) Found 0 VPC's to update Redundant State. 2025-06-29 04:56:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7971af8e]) (logid:d9989750) Found 0 networks to update RvR status. 2025-06-29 04:56:43,108 DEBUG [c.c.n.r.V.UpdateRouterHealthChecksConfigTask] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Found 1 running routers. 2025-06-29 04:56:43,112 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:56:43,113 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Seq 2-3926857400090362227: Routed from 90520738888109 2025-06-29 04:56:43,113 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Seq 2-3926857400090362227: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"true","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-8bf93af8]) (logid:603eee63) External devices are not used. Skipping external device usage collection 2025-06-29 04:56:43,141 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-4e50eee0]) (logid:a80809f7) Started resource counters recalculation periodic task. 2025-06-29 04:56:43,144 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-4e50eee0]) (logid:a80809f7) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 04:56:43,306 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-f0081ff2]) (logid:d66a849f) Backup backup.poll is being called at 2025-06-29T02:56:43+0000 2025-06-29 04:56:43,313 INFO [c.c.h.H.UsageServerMonitorTask] (HA-2:[ctx-ec039ed5]) (logid:5ba2fd7e) checking health of usage server 2025-06-29 04:56:43,314 DEBUG [c.c.h.H.UsageServerMonitorTask] (HA-2:[ctx-ec039ed5]) (logid:5ba2fd7e) usage server running? false, heartbeat: null 2025-06-29 04:56:43,318 WARN [c.c.a.AlertManagerImpl] (HA-2:[ctx-ec039ed5]) (logid:5ba2fd7e) alertType=[13] dataCenterId=[0] podId=[null] clusterId=[null] message=[No usage server process running]. 2025-06-29 04:56:43,320 DEBUG [c.c.a.AlertManagerImpl] (HA-2:[ctx-ec039ed5]) (logid:5ba2fd7e) Have already sent: 1 emails for alert type '13' -- skipping send email 2025-06-29 04:56:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eed46a0]) (logid:e9a47ad3) No inactive management server node found 2025-06-29 04:56:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7eed46a0]) (logid:e9a47ad3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:43,531 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-4e50eee0]) (logid:a80809f7) Finished resource counters recalculation periodic task. 2025-06-29 04:56:43,685 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:[]) (logid:) Seq 2-3926857400090362227: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"monitoringResults":"success","result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:43,685 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Seq 2-3926857400090362227: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:56:43,685 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: success 2025-06-29 04:56:43,688 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Updating data for router health checks for router 94c6bcb2-e08e-4693-9293-c68f7bb6e85b 2025-06-29 04:56:43,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Wait time setting on com.cloud.agent.api.routing.SetMonitorServiceCommand is 1800 seconds 2025-06-29 04:56:43,729 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Seq 2-3926857400090362228: Routed from 90520738888109 2025-06-29 04:56:43,730 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Seq 2-3926857400090362228: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"healthChecksConfig":{"gateways":"gatewaysIps=10.27.0.1 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":"vmName=has,macAddress=1e:00:5d:00:05:95,ip=10.27.0.132,dhcp=true,dns=true;vmName=ptero-node2,macAddress=1e:00:b8:00:05:9c,ip=10.27.0.139,dhcp=true,dns=true;vmName=mgmt-bjnrc,macAddress=1e:00:a6:00:05:99,ip=10.27.0.136,dhcp=true,dns=true;vmName=prod-q97bv,macAddress=1e:00:14:00:05:98,ip=10.27.0.135,dhcp=true,dns=true;vmName=prod-md-0-jw74j-tds8r,macAddress=1e:00:c2:00:05:9b,ip=10.27.0.138,dhcp=true,dns=true;","routerVersion":"templateVersion=Cloudstack Release 4.20.1 Wed May 28 06:10:04 AM UTC 2025,scriptsVersion=794378020e46ba0e608268ece17b62cc "},"reconfigureAfterUpdate":"true","deleteFromProcessedCache":"true","accessDetails":{"router.name":"r-15-VM","router.health.checks.advanced.interval":"10","router.health.checks.enabled":"true","router.health.checks.basic.interval":"3","router.health.checks.excluded":"","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) ManagementServerCollector is running... 2025-06-29 04:56:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) DB properties were already loaded 2025-06-29 04:56:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:56:44,055 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Execution of process [4068245] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:56:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Exit value of process [4068245] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:56:44,056 WARN [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Process [4068245] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:56:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) usage install: null 2025-06-29 04:56:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Sessions active: 0 2025-06-29 04:56:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:56:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068248] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:56:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:56:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068253] for command [/bin/bash -c uname -r ]. 2025-06-29 04:56:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:56:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068254] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:56:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) system memory from /proc: 4887515136 2025-06-29 04:56:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:56:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068259] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:56:44,081 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) free memory from /proc: 352030720 2025-06-29 04:56:44,081 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:56:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068264] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:56:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) used memory from /proc: 990448 2025-06-29 04:56:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:56:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068265] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:56:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:56:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068266] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:56:44,110 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:56:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068267] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:56:44,117 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:56:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068274] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:56:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:56:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Successfully executed process [4068275] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail buffersdirect.capacity, 331040 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail buffersdirect.count, 97 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail buffersdirect.used, 331040 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail buffersmapped.capacity, 0 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail buffersmapped.count, 0 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail buffersmapped.used, 0 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:56:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail jvmuptime, 4863831 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memoryheap.committed, 671088640 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memoryheap.init, 77594624 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memoryheap.usage, 0.23090798184950906 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorynon-heap.committed, 199294976 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorynon-heap.max, -1 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorynon-heap.usage, -1.96234528E8 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorynon-heap.used, 196234528 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.12044808884297521 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22299796714100042 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.Compressed-Class-Space.usage, 0.012401901185512543 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.Metaspace.usage, 0.9929152906359777 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.PS-Eden-Space.usage, 0.3710362408780004 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorytotal.committed, 870383616 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorytotal.init, 85262336 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorytotal.max, 1908932607 2025-06-29 04:56:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail memorytotal.used, 637022304 2025-06-29 04:56:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail threadsdeadlocks, [] 2025-06-29 04:56:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail threadsnew.count, 0 2025-06-29 04:56:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:56:44,132 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-2:[ctx-9f0ac906]) (logid:8f633ecf) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":231069232,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4863759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":636248592,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":440787776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":352030720,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.26,0.21],"systemCyclesUsage":[66425683,53844707,4590476793],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:56:44,132 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 161, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":231069232,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4863759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":636248592,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":440787776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":352030720,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.26,0.21],"systemCyclesUsage":[66425683,53844707,4590476793],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:56:44,132 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:56:44,138 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-35:[ctx-08d905f7]) (logid:41f047fa) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":231069232,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4863759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":636248592,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":440787776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":352030720,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.26,0.21],"systemCyclesUsage":[66425683,53844707,4590476793],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:56:44,138 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:56:44,139 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-2:[ctx-8527a7d6]) (logid:dfbd6e98) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 161, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:56:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":231069232,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4863759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.25,"totalInit":85262336,"totalUsed":636248592,"totalCommitted":870318080,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":440787776,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":352030720,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.25,0.26,0.21],"systemCyclesUsage":[66425683,53844707,4590476793],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:56:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0ca1dfe]) (logid:6e1c1a66) No inactive management server node found 2025-06-29 04:56:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0ca1dfe]) (logid:6e1c1a66) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:45,164 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:[]) (logid:) Seq 2-3926857400090362228: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.60.241, file: monitor_service.json.727dc66f-7409-4f03-9d5a-d91ab1d18e61","null - success: ","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:45,164 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-a77a8552]) (logid:823e3ea8) Seq 2-3926857400090362228: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GroupAnswer } } 2025-06-29 04:56:45,166 INFO [c.c.n.r.V.FetchRouterHealthChecksResultTask] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found 1 running routers. Fetching, analysing and updating DB for the health checks. 2025-06-29 04:56:45,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Wait time setting on com.cloud.agent.api.routing.GetRouterMonitorResultsCommand is 1800 seconds 2025-06-29 04:56:45,170 DEBUG [c.c.a.m.ClusteredAgentAttache] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Seq 2-3926857400090362229: Routed from 90520738888109 2025-06-29 04:56:45,171 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Seq 2-3926857400090362229: Sending { Cmd , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsCommand":{"performFreshChecks":"false","validateBasicTestsOnly":"false","accessDetails":{"router.name":"r-15-VM","router.ip":"169.254.60.241"},"wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:45,422 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8834: Seq 7-8834: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:45,432 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8834 2025-06-29 04:56:45,589 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:[]) (logid:) Seq 2-3926857400090362229: Processing: { Ans: , MgmtId: 90520738888109, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":[],"monitoringResults":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "9.711503982543945", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "21.678447723388672", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "33.28895568847656", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "45.98069190979004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165641806", "lastRunDuration": "48.45166206359863", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165641879", "lastRunDuration": "8070.453405380249", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165649994", "lastRunDuration": "63.30084800720215", "message": "Memory Usage within limits with current at 50.8868%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165650058", "lastRunDuration": "3235.6576919555664", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165653351", "lastRunDuration": "50.280094146728516", "message": "Sufficient free space is 2571.20703125 MB"}, "lastRun": {"start": "2025-06-29 02:54:01.635944", "end": "2025-06-29 02:54:13.401983", "duration": "11.766038656234741"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751165401300", "lastRunDuration": "44.92807388305664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751165401346", "lastRunDuration": "59.64183807373047", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751165401532", "lastRunDuration": "41.38660430908203", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751165401686", "lastRunDuration": "41.90802574157715", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:50:01.298625", "end": "2025-06-29 02:50:01.771209", "duration": "0.47258448600769043"}}}","result":"true","details":"{"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "9.711503982543945", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "21.678447723388672", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "33.28895568847656", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "45.98069190979004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165641806", "lastRunDuration": "48.45166206359863", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165641879", "lastRunDuration": "8070.453405380249", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165649994", "lastRunDuration": "63.30084800720215", "message": "Memory Usage within limits with current at 50.8868%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165650058", "lastRunDuration": "3235.6576919555664", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165653351", "lastRunDuration": "50.280094146728516", "message": "Sufficient free space is 2571.20703125 MB"}, "lastRun": {"start": "2025-06-29 02:54:01.635944", "end": "2025-06-29 02:54:13.401983", "duration": "11.766038656234741"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751165401300", "lastRunDuration": "44.92807388305664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751165401346", "lastRunDuration": "59.64183807373047", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751165401532", "lastRunDuration": "41.38660430908203", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751165401686", "lastRunDuration": "41.90802574157715", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:50:01.298625", "end": "2025-06-29 02:50:01.771209", "duration": "0.47258448600769043"}}}","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:45,589 DEBUG [c.c.a.t.Request] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Seq 2-3926857400090362229: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } 2025-06-29 04:56:45,589 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Details from executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "9.711503982543945", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "21.678447723388672", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "33.28895568847656", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "45.98069190979004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165641806", "lastRunDuration": "48.45166206359863", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165641879", "lastRunDuration": "8070.453405380249", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165649994", "lastRunDuration": "63.30084800720215", "message": "Memory Usage within limits with current at 50.8868%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165650058", "lastRunDuration": "3235.6576919555664", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165653351", "lastRunDuration": "50.280094146728516", "message": "Sufficient free space is 2571.20703125 MB"}, "lastRun": {"start": "2025-06-29 02:54:01.635944", "end": "2025-06-29 02:54:13.401983", "duration": "11.766038656234741"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751165401300", "lastRunDuration": "44.92807388305664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751165401346", "lastRunDuration": "59.64183807373047", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751165401532", "lastRunDuration": "41.38660430908203", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751165401686", "lastRunDuration": "41.90802574157715", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:50:01.298625", "end": "2025-06-29 02:50:01.771209", "duration": "0.47258448600769043"}}} 2025-06-29 04:56:45,620 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Parsing and updating DB health check data for router: VM instance {"id":15,"instanceName":"r-15-VM","state":"Running","type":"DomainRouter","uuid":"94c6bcb2-e08e-4693-9293-c68f7bb6e85b"} with data: {"basic":{"dhcp.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "9.711503982543945", "message": "service is running"}, "loadbalancing.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "21.678447723388672", "message": "service is running"}, "ssh.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "33.28895568847656", "message": "service is running"}, "webserver.service": {"success": "true", "lastUpdate": "1751165641637", "lastRunDuration": "45.98069190979004", "message": "service is running"}, "router_version_check.py": {"success": "true", "lastUpdate": "1751165641806", "lastRunDuration": "48.45166206359863", "message": "Template and scripts version match successful"}, "gateways_check.py": {"success": "true", "lastUpdate": "1751165641879", "lastRunDuration": "8070.453405380249", "message": "All 2 gateways are reachable via ping"}, "memory_usage_check.py": {"success": "true", "lastUpdate": "1751165649994", "lastRunDuration": "63.30084800720215", "message": "Memory Usage within limits with current at 50.8868%"}, "cpu_usage_check.py": {"success": "true", "lastUpdate": "1751165650058", "lastRunDuration": "3235.6576919555664", "message": "CPU Usage within limits with current at 0.0%"}, "disk_space_check.py": {"success": "true", "lastUpdate": "1751165653351", "lastRunDuration": "50.280094146728516", "message": "Sufficient free space is 2571.20703125 MB"}, "lastRun": {"start": "2025-06-29 02:54:01.635944", "end": "2025-06-29 02:54:13.401983", "duration": "11.766038656234741"}},"advanced":{"haproxy_check.py": {"success": "true", "lastUpdate": "1751165401300", "lastRunDuration": "44.92807388305664", "message": "No data provided to check, skipping"}, "iptables_check.py": {"success": "true", "lastUpdate": "1751165401346", "lastRunDuration": "59.64183807373047", "message": "No portforwarding rules provided to check, skipping"}, "dns_check.py": {"success": "true", "lastUpdate": "1751165401532", "lastRunDuration": "41.38660430908203", "message": "All 5 VMs are present in /etc/hosts"}, "dhcp_check.py": {"success": "true", "lastUpdate": "1751165401686", "lastRunDuration": "41.90802574157715", "message": "All 5 VMs are present in dhcphosts.txt"}, "lastRun": {"start": "2025-06-29 02:50:01.298625", "end": "2025-06-29 02:50:01.771209", "duration": "0.47258448600769043"}}} 2025-06-29 04:56:45,622 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found check types executed on VR basic, start: 2025-06-29 02:54:01.635944, end: 2025-06-29 02:54:13.401983, duration: 11.766038656234741 2025-06-29 04:56:45,631 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2ce70a59- check type: basic,check name: dhcp.service, check result: true, check last update: Sun Jun 29 04:54:01 CEST 2025, details: service is running which took running duration (ms) 9.711503982543945 2025-06-29 04:56:45,643 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4da2ac0a- check type: basic,check name: loadbalancing.service, check result: true, check last update: Sun Jun 29 04:54:01 CEST 2025, details: service is running which took running duration (ms) 21.678447723388672 2025-06-29 04:56:45,652 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@4e760e18- check type: basic,check name: ssh.service, check result: true, check last update: Sun Jun 29 04:54:01 CEST 2025, details: service is running which took running duration (ms) 33.28895568847656 2025-06-29 04:56:45,663 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@30ad99e6- check type: basic,check name: webserver.service, check result: true, check last update: Sun Jun 29 04:54:01 CEST 2025, details: service is running which took running duration (ms) 45.98069190979004 2025-06-29 04:56:45,674 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@248cf868- check type: basic,check name: router_version_check.py, check result: true, check last update: Sun Jun 29 04:54:01 CEST 2025, details: Template and scripts version match successful which took running duration (ms) 48.45166206359863 2025-06-29 04:56:45,684 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7c16c20f- check type: basic,check name: gateways_check.py, check result: true, check last update: Sun Jun 29 04:54:01 CEST 2025, details: All 2 gateways are reachable via ping which took running duration (ms) 8070.453405380249 2025-06-29 04:56:45,695 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@2ee055b- check type: basic,check name: memory_usage_check.py, check result: true, check last update: Sun Jun 29 04:54:09 CEST 2025, details: Memory Usage within limits with current at 50.8868% which took running duration (ms) 63.30084800720215 2025-06-29 04:56:45,704 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@5e50de49- check type: basic,check name: cpu_usage_check.py, check result: true, check last update: Sun Jun 29 04:54:10 CEST 2025, details: CPU Usage within limits with current at 0.0% which took running duration (ms) 3235.6576919555664 2025-06-29 04:56:45,715 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@1d5c2253- check type: basic,check name: disk_space_check.py, check result: true, check last update: Sun Jun 29 04:54:13 CEST 2025, details: Sufficient free space is 2571.20703125 MB which took running duration (ms) 50.280094146728516 2025-06-29 04:56:45,715 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found check types executed on VR advanced, start: 2025-06-29 02:50:01.298625, end: 2025-06-29 02:50:01.771209, duration: 0.47258448600769043 2025-06-29 04:56:45,726 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@57909d4d- check type: advanced,check name: haproxy_check.py, check result: true, check last update: Sun Jun 29 04:50:01 CEST 2025, details: No data provided to check, skipping which took running duration (ms) 44.92807388305664 2025-06-29 04:56:45,737 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@25a69e39- check type: advanced,check name: iptables_check.py, check result: true, check last update: Sun Jun 29 04:50:01 CEST 2025, details: No portforwarding rules provided to check, skipping which took running duration (ms) 59.64183807373047 2025-06-29 04:56:45,747 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@7ddcae52- check type: advanced,check name: dns_check.py, check result: true, check last update: Sun Jun 29 04:50:01 CEST 2025, details: All 5 VMs are present in /etc/hosts which took running duration (ms) 41.38660430908203 2025-06-29 04:56:45,756 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:[ctx-134a4d8f]) (logid:4630e36f) Found health check com.cloud.network.dao.RouterHealthCheckResultVO@16e4500- check type: advanced,check name: dhcp_check.py, check result: true, check last update: Sun Jun 29 04:50:01 CEST 2025, details: All 5 VMs are present in dhcphosts.txt which took running duration (ms) 41.90802574157715 2025-06-29 04:56:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:56:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:56:46,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d83a2db]) (logid:3d990356) No inactive management server node found 2025-06-29 04:56:46,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8d83a2db]) (logid:3d990356) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:56:46,643 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:56:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:56:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:56:46,979 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-c342362f]) (logid:5a8fe8b8) HA health check task is running... 2025-06-29 04:56:47,083 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:56:47,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:56:47,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:56:47,088 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:56:47,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ed3063d]) (logid:d42bc51c) No inactive management server node found 2025-06-29 04:56:47,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ed3063d]) (logid:d42bc51c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:48,062 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-2:[ctx-8b78c60f]) (logid:c6324a4a) Backup Sync Task is not enabled in zone [Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"}]. Skipping this zone! 2025-06-29 04:56:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6a64ac57]) (logid:5dbf969c) VM scheduler.poll is being called at 2025-06-29T02:57:00+0000 2025-06-29 04:56:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6a64ac57]) (logid:5dbf969c) Got 0 scheduled jobs to be executed at 2025-06-29T02:57:00+0000 2025-06-29 04:56:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-6a64ac57]) (logid:5dbf969c) Cleaned up 0 VM scheduled job entries 2025-06-29 04:56:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-8cde8213]) (logid:bdec1ffe) VM Operation Thread Running 2025-06-29 04:56:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-40289188]) (logid:35c344c7) ClusterDRS.poll is being called at 2025-06-29T02:57:00+0000 2025-06-29 04:56:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-40289188]) (logid:35c344c7) Removed 0 old drs migration plans 2025-06-29 04:56:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29eb5426]) (logid:edf2890e) No inactive management server node found 2025-06-29 04:56:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-29eb5426]) (logid:edf2890e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:50,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd71d0e3]) (logid:1a158c09) No inactive management server node found 2025-06-29 04:56:50,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bd71d0e3]) (logid:1a158c09) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:50,980 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-0eb286a5]) (logid:dcdcdd37) HA health check task is running... 2025-06-29 04:56:52,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77d02e9f]) (logid:6f4bd77f) No inactive management server node found 2025-06-29 04:56:52,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77d02e9f]) (logid:6f4bd77f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:56:53,150 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-5c376c46]) (logid:c6c83460) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:56:53,161 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:[ctx-5c376c46]) (logid:c6c83460) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status 2025-06-29 04:56:53,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f54dfda8]) (logid:fa484e1f) No inactive management server node found 2025-06-29 04:56:53,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f54dfda8]) (logid:fa484e1f) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:56:54,982 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-c0c1f7ba]) (logid:55f33ff0) HA health check task is running... 2025-06-29 04:56:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7768317f]) (logid:f9eded70) No inactive management server node found 2025-06-29 04:56:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7768317f]) (logid:f9eded70) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:55,422 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8836: Seq 7-8836: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:56:55,431 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8836 2025-06-29 04:56:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fb724b3]) (logid:8d819676) No inactive management server node found 2025-06-29 04:56:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fb724b3]) (logid:8d819676) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-44a799c7]) (logid:20648ebb) Enabled non-edge zones available for scan: 1 2025-06-29 04:56:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-44a799c7]) (logid:20648ebb) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:56:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-44a799c7]) (logid:20648ebb) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:56:58,098 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:56:58,101 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Seq 2-4725683383995203915: Routed from 90520738888109 2025-06-29 04:56:58,144 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Seq 1-4725683383995203915: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:56:58,144 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool 27ea62dc-ded2-4a25-9115-277c981f33fa from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '27ea62dc-ded2-4a25-9115-277c981f33fa' 2025-06-29 04:56:58,172 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Wait time setting on com.cloud.agent.api.GetVolumeStatsCommand is 60 seconds 2025-06-29 04:56:58,173 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Seq 2-3926857400090362230: Routed from 90520738888109 2025-06-29 04:56:58,217 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Seq 2-3926857400090362230: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVolumeStatsAnswer } } 2025-06-29 04:56:58,217 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-2e1e78db]) (logid:a1fa1ebd) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: Can't get vm disk stats: Could not fetch storage pool d997caf8-e98c-498c-aac2-3016f6ae2f5d from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'd997caf8-e98c-498c-aac2-3016f6ae2f5d' 2025-06-29 04:56:58,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dca32005]) (logid:b4e9901b) Enabled non-edge zones available for scan: 1 2025-06-29 04:56:58,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dca32005]) (logid:b4e9901b) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:56:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35b7a52c]) (logid:e5af78f1) No inactive management server node found 2025-06-29 04:56:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-35b7a52c]) (logid:e5af78f1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:56:58,983 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-cb5ac4a4]) (logid:699859d0) HA health check task is running... 2025-06-29 04:56:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-507fc158]) (logid:87c3c20e) No inactive management server node found 2025-06-29 04:56:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-507fc158]) (logid:87c3c20e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0a4f69b]) (logid:412e583a) No inactive management server node found 2025-06-29 04:57:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e0a4f69b]) (logid:412e583a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:02,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-afb006c0]) (logid:d9fd2a24) No inactive management server node found 2025-06-29 04:57:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-afb006c0]) (logid:d9fd2a24) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:02,984 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-6c56c57e]) (logid:0cfad32f) HA health check task is running... 2025-06-29 04:57:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9430bef]) (logid:3be6d940) No inactive management server node found 2025-06-29 04:57:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e9430bef]) (logid:3be6d940) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:05,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8837: Seq 7-8837: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:57:05,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8837 2025-06-29 04:57:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-296e261f]) (logid:25d68688) No inactive management server node found 2025-06-29 04:57:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-296e261f]) (logid:25d68688) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:06,985 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-37f864e1]) (logid:98fd498e) HA health check task is running... 2025-06-29 04:57:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-551b63fe]) (logid:239e2b2c) No inactive management server node found 2025-06-29 04:57:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-551b63fe]) (logid:239e2b2c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a51819a]) (logid:96e2098c) No inactive management server node found 2025-06-29 04:57:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9a51819a]) (logid:96e2098c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e976529]) (logid:d8834ec5) No inactive management server node found 2025-06-29 04:57:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7e976529]) (logid:d8834ec5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:10,986 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bb19ddf6]) (logid:3b0dc5b8) HA health check task is running... 2025-06-29 04:57:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77f090f5]) (logid:d290cd89) No inactive management server node found 2025-06-29 04:57:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-77f090f5]) (logid:d290cd89) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:13,005 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-51b83355]) (logid:bea6f0b9) Checking if any host reservation can be released ... 2025-06-29 04:57:13,010 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-51b83355]) (logid:bea6f0b9) Cannot release reservation, Found 3 VMs Running on host Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-51b83355]) (logid:bea6f0b9) Cannot release reservation, Found 5 VMs Running on host Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:13,013 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:[ctx-51b83355]) (logid:bea6f0b9) Done running HostReservationReleaseChecker ... 2025-06-29 04:57:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-51352a76]) (logid:29875ed2) Found 0 routers to update status. 2025-06-29 04:57:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-51352a76]) (logid:29875ed2) Found 0 VPC's to update Redundant State. 2025-06-29 04:57:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-51352a76]) (logid:29875ed2) Found 0 networks to update RvR status. 2025-06-29 04:57:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Running Capacity Checker ... 2025-06-29 04:57:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) recalculating system capacity 2025-06-29 04:57:13,167 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Executing cpu/ram capacity update 2025-06-29 04:57:13,172 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) Found 3 VMs on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:13,173 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:13,180 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) Found 5 VM, not running on Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:13,186 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpuCore: 18 reservedCpuCore: 0 2025-06-29 04:57:13,187 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedCpu: 52000 reservedCpu: 0 2025-06-29 04:57:13,187 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} usedMem: (36.00 GB) 38654705664 reservedMem: (0 bytes) 0 2025-06-29 04:57:13,199 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) Found 5 VMs on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:13,200 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) Found 0 VMs are Migrating from Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:13,207 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) Found 0 VM, not running on Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpuCore: 9 reservedCpuCore: 0 2025-06-29 04:57:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) No need to calibrate cpu capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedCpu: 13500 reservedCpu: 0 2025-06-29 04:57:13,209 DEBUG [c.c.c.CapacityManagerImpl] (pool-44-thread-1:[]) (logid:) No need to calibrate memory capacity, host:Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} usedMem: (14.00 GB) 15032385536 reservedMem: (0 bytes) 0 2025-06-29 04:57:13,219 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Done executing cpu/ram capacity update 2025-06-29 04:57:13,219 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Executing storage capacity update 2025-06-29 04:57:13,227 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Found storage pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:57:13,227 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (55.8441 TB) 61401196134400 2025-06-29 04:57:13,227 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"} is (111.6881 TB) 122802392268800 2025-06-29 04:57:13,229 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Successfully set Capacity - (111.6881 TB) 122802392268800 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":4,"name":"nfs","poolType":"NetworkFilesystem","uuid":"10805e0a-2205-3158-993f-9350b08b9137"}, PodId 1 2025-06-29 04:57:13,236 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Found storage pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} of type NetworkFilesystem with overprovisioning factor 2 2025-06-29 04:57:13,237 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (1.2900 TB) 1418370088960 2025-06-29 04:57:13,237 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"} is (2.5800 TB) 2836740177920 2025-06-29 04:57:13,241 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Successfully set Capacity - (2.5800 TB) 2836740177920 for capacity type - 3 , DataCenterId - 1, Pool - StoragePool {"id":5,"name":"nfs-ssd","poolType":"NetworkFilesystem","uuid":"80a947f2-ab02-3d8c-b1c0-7ba964ae0e80"}, PodId 1 2025-06-29 04:57:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Found storage pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:57:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (850.65 GB) 913383583744 2025-06-29 04:57:13,249 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"} is (1.6614 TB) 1826767167488 2025-06-29 04:57:13,253 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Successfully set Capacity - (1.6614 TB) 1826767167488 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":12,"name":"cs-compute1.cloud.redacted.net-local-27ea62dc","poolType":"Filesystem","uuid":"27ea62dc-ded2-4a25-9115-277c981f33fa"}, PodId 1 2025-06-29 04:57:13,260 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Found storage pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} of type Filesystem with overprovisioning factor 2 2025-06-29 04:57:13,260 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity calculated is 2 * (153.39 GB) 164705353728 2025-06-29 04:57:13,260 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Total over provisioned capacity of the pool StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"} is (306.79 GB) 329410707456 2025-06-29 04:57:13,264 DEBUG [c.c.s.StorageManagerImpl] (pool-45-thread-1:[]) (logid:) Successfully set Capacity - (306.79 GB) 329410707456 for capacity type - 9 , DataCenterId - 1, Pool - StoragePool {"id":13,"name":"cs-compute2.cloud.redacted.net-local-d997caf8","poolType":"Filesystem","uuid":"d997caf8-e98c-498c-aac2-3016f6ae2f5d"}, PodId 1 2025-06-29 04:57:13,264 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Done executing storage capacity update 2025-06-29 04:57:13,264 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Executing capacity updates for public ip and Vlans 2025-06-29 04:57:13,284 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Done capacity updates for public ip and Vlans 2025-06-29 04:57:13,284 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Executing capacity updates for private ip 2025-06-29 04:57:13,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Done executing capacity updates for private ip 2025-06-29 04:57:13,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Done recalculating system capacity 2025-06-29 04:57:13,315 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:[ctx-0b6c2c71]) (logid:c2dd9144) Done running Capacity Checker ... 2025-06-29 04:57:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42b4c6f9]) (logid:56dda650) No inactive management server node found 2025-06-29 04:57:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42b4c6f9]) (logid:56dda650) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5697e0ea]) (logid:019ee9d2) No inactive management server node found 2025-06-29 04:57:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5697e0ea]) (logid:019ee9d2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:14,987 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-77653127]) (logid:d84eeb51) HA health check task is running... 2025-06-29 04:57:15,423 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8838: Seq 7-8838: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:57:15,434 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8838 2025-06-29 04:57:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d987b435]) (logid:537314b9) No inactive management server node found 2025-06-29 04:57:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d987b435]) (logid:537314b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239b9a3f]) (logid:67b279ac) No inactive management server node found 2025-06-29 04:57:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-239b9a3f]) (logid:67b279ac) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:18,102 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-74f0bfe7]) (logid:7118e479) AutoScaling Monitor is running... 2025-06-29 04:57:18,185 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-0a70c672]) (logid:3a6b0b35) Starting to remove all volume_stats rows older than [Sat Jun 28 16:57:18 CEST 2025]. 2025-06-29 04:57:18,186 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-0a70c672]) (logid:3a6b0b35) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:57:18 CEST 2025]. 2025-06-29 04:57:18,988 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-3:[ctx-bd1f4148]) (logid:045aa083) HA health check task is running... 2025-06-29 04:57:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-beefe283]) (logid:2bafe57e) No inactive management server node found 2025-06-29 04:57:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-beefe283]) (logid:2bafe57e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:19,458 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a92be3c2]) (logid:2665af51) Starting to remove all vm_stats rows older than [Sat Jun 28 16:57:19 CEST 2025]. 2025-06-29 04:57:19,496 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-a92be3c2]) (logid:2665af51) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:57:19 CEST 2025]. 2025-06-29 04:57:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2058874c]) (logid:4a844b48) No inactive management server node found 2025-06-29 04:57:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2058874c]) (logid:4a844b48) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49fb11ca]) (logid:a4a287eb) No inactive management server node found 2025-06-29 04:57:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49fb11ca]) (logid:a4a287eb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:22,886 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:57:22,888 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:57:22,889 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) Seq 2-4725683383995203916: Routed from 90520738888109 2025-06-29 04:57:22,932 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) Seq 1-4725683383995203916: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:57:22,933 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:57:22,934 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) Seq 2-3926857400090362231: Routed from 90520738888109 2025-06-29 04:57:22,978 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-699c03e1]) (logid:320ca1b0) Seq 2-3926857400090362231: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:57:22,990 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5ea4da1d]) (logid:02050e09) HA health check task is running... 2025-06-29 04:57:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6c3d161]) (logid:095383ee) No inactive management server node found 2025-06-29 04:57:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a6c3d161]) (logid:095383ee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:25,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f97b7dad]) (logid:ae806714) No inactive management server node found 2025-06-29 04:57:25,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f97b7dad]) (logid:ae806714) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:25,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8839: Seq 7-8839: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:57:25,434 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8839 2025-06-29 04:57:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42037855]) (logid:b19e07c1) No inactive management server node found 2025-06-29 04:57:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-42037855]) (logid:b19e07c1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:26,991 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c8ddf8cb]) (logid:1037c5bd) HA health check task is running... 2025-06-29 04:57:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a8e847f6]) (logid:016a479c) Enabled non-edge zones available for scan: 1 2025-06-29 04:57:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a8e847f6]) (logid:016a479c) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:57:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a8e847f6]) (logid:016a479c) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:57:28,273 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-723e9c2a]) (logid:879e1b5c) Enabled non-edge zones available for scan: 1 2025-06-29 04:57:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-723e9c2a]) (logid:879e1b5c) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:57:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49a209a9]) (logid:23069a3e) No inactive management server node found 2025-06-29 04:57:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49a209a9]) (logid:23069a3e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-159db491]) (logid:5a7bec34) No inactive management server node found 2025-06-29 04:57:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-159db491]) (logid:5a7bec34) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:30,965 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:57:30,969 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:57:30,970 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) Seq 2-4725683383995203917: Routed from 90520738888109 2025-06-29 04:57:30,992 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7acbba90]) (logid:a83f2b06) HA health check task is running... 2025-06-29 04:57:31,059 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) Seq 1-4725683383995203917: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:57:31,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:57:31,094 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) Seq 2-3926857400090362232: Routed from 90520738888109 2025-06-29 04:57:31,188 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-33c493d5]) (logid:b8d32c62) Seq 2-3926857400090362232: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:57:31,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d255faaf]) (logid:584209f2) No inactive management server node found 2025-06-29 04:57:31,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d255faaf]) (logid:584209f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:32,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9efaccd4]) (logid:ec6eb1ac) No inactive management server node found 2025-06-29 04:57:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9efaccd4]) (logid:ec6eb1ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:33,316 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) StorageCollector is running... 2025-06-29 04:57:33,324 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:57:33,324 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:57:33,324 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:57:33,325 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-3938116399158788184: Routed from 90520738888109 2025-06-29 04:57:33,404 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 4-3938116399158788184: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:57:33,408 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:57:33,408 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:57:33,408 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:57:33,409 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-4725683383995203918: Routed from 90520738888109 2025-06-29 04:57:33,416 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 1-4725683383995203918: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:57:33,427 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:57:33,427 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:57:33,427 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:57:33,428 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-3926857400090362233: Routed from 90520738888109 2025-06-29 04:57:33,439 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-3926857400090362233: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:57:33,449 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:57:33,449 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:57:33,449 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:57:33,450 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-4725683383995203919: Routed from 90520738888109 2025-06-29 04:57:33,534 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 1-4725683383995203919: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:57:33,536 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:57:33,536 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:57:33,536 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:57:33,537 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-3926857400090362234: Routed from 90520738888109 2025-06-29 04:57:33,581 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-6d444150]) (logid:50830710) Seq 2-3926857400090362234: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:57:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-807e91ed]) (logid:1dc332dd) No inactive management server node found 2025-06-29 04:57:34,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-807e91ed]) (logid:1dc332dd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:34,993 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-f4d033e9]) (logid:3b875320) HA health check task is running... 2025-06-29 04:57:35,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8840: Seq 7-8840: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:57:35,434 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8840 2025-06-29 04:57:35,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60e5a0b8]) (logid:f51858cd) No inactive management server node found 2025-06-29 04:57:35,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60e5a0b8]) (logid:f51858cd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcdd2c1a]) (logid:98ed0e89) No inactive management server node found 2025-06-29 04:57:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fcdd2c1a]) (logid:98ed0e89) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46c031b4]) (logid:48751e61) No inactive management server node found 2025-06-29 04:57:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-46c031b4]) (logid:48751e61) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:38,994 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-d41042fa]) (logid:9febd678) HA health check task is running... 2025-06-29 04:57:40,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-610ab91b]) (logid:a5b57951) No inactive management server node found 2025-06-29 04:57:40,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-610ab91b]) (logid:a5b57951) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22f1b8ec]) (logid:cf3d318c) No inactive management server node found 2025-06-29 04:57:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-22f1b8ec]) (logid:cf3d318c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:42,995 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-33f4cfe4]) (logid:8e556d1e) HA health check task is running... 2025-06-29 04:57:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-3b173f3b]) (logid:a0886feb) DbCollector is running... 2025-06-29 04:57:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-53320b14]) (logid:4033c5ff) Found 0 routers to update status. 2025-06-29 04:57:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-53320b14]) (logid:4033c5ff) Found 0 VPC's to update Redundant State. 2025-06-29 04:57:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-53320b14]) (logid:4033c5ff) Found 0 networks to update RvR status. 2025-06-29 04:57:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e157ae98]) (logid:30412dee) No inactive management server node found 2025-06-29 04:57:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e157ae98]) (logid:30412dee) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) ManagementServerCollector is running... 2025-06-29 04:57:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) DB properties were already loaded 2025-06-29 04:57:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:57:44,055 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Execution of process [4068289] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:57:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Exit value of process [4068289] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:57:44,056 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Process [4068289] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:57:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) usage install: null 2025-06-29 04:57:44,056 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Sessions active: 0 2025-06-29 04:57:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:57:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068292] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:57:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:57:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068297] for command [/bin/bash -c uname -r ]. 2025-06-29 04:57:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:57:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068298] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:57:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) system memory from /proc: 4887515136 2025-06-29 04:57:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:57:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068303] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:57:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) free memory from /proc: 348684288 2025-06-29 04:57:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:57:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068308] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:57:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) used memory from /proc: 990448 2025-06-29 04:57:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:57:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068309] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:57:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:57:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068310] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:57:44,109 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:57:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068311] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:57:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:57:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068318] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:57:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:57:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Successfully executed process [4068319] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail buffersdirect.capacity, 331040 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail buffersdirect.count, 97 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail buffersdirect.used, 331040 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail buffersmapped.capacity, 0 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail buffersmapped.count, 0 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail buffersmapped.used, 0 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail jvmuptime, 4923830 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memoryheap.committed, 671088640 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memoryheap.init, 77594624 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memoryheap.usage, 0.25061258946235154 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorynon-heap.committed, 199360512 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorynon-heap.max, -1 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorynon-heap.usage, -1.9630944E8 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorynon-heap.used, 196309440 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.12067407024793389 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2232354117372613 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.Compressed-Class-Space.usage, 0.012402929365634918 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.Metaspace.usage, 0.9930357028505562 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.PS-Eden-Space.usage, 0.4294125438221496 2025-06-29 04:57:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorytotal.committed, 870449152 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorytotal.init, 85262336 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorytotal.max, 1908932607 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail memorytotal.used, 674711984 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail threadsdeadlocks, [] 2025-06-29 04:57:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail threadsnew.count, 0 2025-06-29 04:57:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:57:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-c8cbbd39]) (logid:dcca3a22) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":193435216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4923759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.23,"totalInit":85262336,"totalUsed":673961760,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":478402544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348684288,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.23,0.24,0.2],"systemCyclesUsage":[66426017,53845010,4590499907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:57:44,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 163, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":193435216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4923759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.23,"totalInit":85262336,"totalUsed":673961760,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":478402544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348684288,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.23,0.24,0.2],"systemCyclesUsage":[66426017,53845010,4590499907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:57:44,131 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:57:44,137 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-35:[ctx-f15e8874]) (logid:3defe3c9) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":193435216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4923759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.23,"totalInit":85262336,"totalUsed":673961760,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":478402544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348684288,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.23,0.24,0.2],"systemCyclesUsage":[66426017,53845010,4590499907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:57:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 6 ms 2025-06-29 04:57:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:[ctx-2eb8dcfc]) (logid:bf6f8b11) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 6ms. agent: 0, pdu seq: 163, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:57:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":193435216,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4923759,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.23,"totalInit":85262336,"totalUsed":673961760,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":478402544,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348684288,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.23,0.24,0.2],"systemCyclesUsage":[66426017,53845010,4590499907],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:57:44,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd6b752d]) (logid:ec467c4e) No inactive management server node found 2025-06-29 04:57:44,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd6b752d]) (logid:ec467c4e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:45,424 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8841: Seq 7-8841: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:57:45,435 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8841 2025-06-29 04:57:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:57:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:57:46,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9baa0950]) (logid:472d4ffb) No inactive management server node found 2025-06-29 04:57:46,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9baa0950]) (logid:472d4ffb) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:57:46,649 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-11:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:57:46,996 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a84489cf]) (logid:30b08b6a) HA health check task is running... 2025-06-29 04:57:47,092 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:47,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:57:47,098 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:57:47,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4641afa4]) (logid:747e4665) No inactive management server node found 2025-06-29 04:57:47,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4641afa4]) (logid:747e4665) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-22c90924]) (logid:98db3d87) Resetting hosts suitable for reconnect 2025-06-29 04:57:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-22c90924]) (logid:98db3d87) Completed resetting hosts suitable for reconnect 2025-06-29 04:57:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-22c90924]) (logid:98db3d87) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:57:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-22c90924]) (logid:98db3d87) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:57:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-22c90924]) (logid:98db3d87) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:57:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-22c90924]) (logid:98db3d87) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:57:48,157 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b89b7961]) (logid:73601566) VM scheduler.poll is being called at 2025-06-29T02:58:00+0000 2025-06-29 04:57:48,161 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b89b7961]) (logid:73601566) Got 0 scheduled jobs to be executed at 2025-06-29T02:58:00+0000 2025-06-29 04:57:48,163 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-b89b7961]) (logid:73601566) Cleaned up 0 VM scheduled job entries 2025-06-29 04:57:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ec10b3d0]) (logid:c435c6f5) VM Operation Thread Running 2025-06-29 04:57:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e4b03b2e]) (logid:a3d5962a) ClusterDRS.poll is being called at 2025-06-29T02:58:00+0000 2025-06-29 04:57:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-e4b03b2e]) (logid:a3d5962a) Removed 0 old drs migration plans 2025-06-29 04:57:49,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f83f4eea]) (logid:b24f2485) No inactive management server node found 2025-06-29 04:57:49,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f83f4eea]) (logid:b24f2485) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-679ab40c]) (logid:e1ff0689) No inactive management server node found 2025-06-29 04:57:50,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-679ab40c]) (logid:e1ff0689) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:50,998 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a60144ac]) (logid:b2d27917) HA health check task is running... 2025-06-29 04:57:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44bafe73]) (logid:1f07f3ac) No inactive management server node found 2025-06-29 04:57:52,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-44bafe73]) (logid:1f07f3ac) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8f3c2c9]) (logid:a75ebcdd) No inactive management server node found 2025-06-29 04:57:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f8f3c2c9]) (logid:a75ebcdd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:54,999 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-6e2de95c]) (logid:237c1f1f) HA health check task is running... 2025-06-29 04:57:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a738ea5e]) (logid:4cca07b9) No inactive management server node found 2025-06-29 04:57:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a738ea5e]) (logid:4cca07b9) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:55,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8843: Seq 7-8843: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:57:55,435 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8843 2025-06-29 04:57:56,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b86ef69f]) (logid:0ce338b6) No inactive management server node found 2025-06-29 04:57:56,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b86ef69f]) (logid:0ce338b6) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-84ee3143]) (logid:c374c301) Enabled non-edge zones available for scan: 1 2025-06-29 04:57:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-84ee3143]) (logid:c374c301) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:57:58,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-84ee3143]) (logid:c374c301) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:57:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a4d70019]) (logid:e2aec1b3) Enabled non-edge zones available for scan: 1 2025-06-29 04:57:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a4d70019]) (logid:e2aec1b3) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:57:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91339d58]) (logid:0e20af31) No inactive management server node found 2025-06-29 04:57:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-91339d58]) (logid:0e20af31) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:57:59,000 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2054b6db]) (logid:9e2504e5) HA health check task is running... 2025-06-29 04:57:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51cd5720]) (logid:bdac760e) No inactive management server node found 2025-06-29 04:57:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51cd5720]) (logid:bdac760e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac423290]) (logid:832a7b2a) No inactive management server node found 2025-06-29 04:58:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ac423290]) (logid:832a7b2a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:02,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1085ac88]) (logid:f8aeb7c8) No inactive management server node found 2025-06-29 04:58:02,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1085ac88]) (logid:f8aeb7c8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:03,002 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-91cfd9ab]) (logid:86bf33bb) HA health check task is running... 2025-06-29 04:58:04,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0241b56d]) (logid:5377a09c) No inactive management server node found 2025-06-29 04:58:04,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0241b56d]) (logid:5377a09c) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:05,426 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-8844: Seq 7-8844: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:58:05,436 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) SeqA 7-: Sending 8844 2025-06-29 04:58:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-019ceb3c]) (logid:2ebbd6ff) No inactive management server node found 2025-06-29 04:58:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-019ceb3c]) (logid:2ebbd6ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:07,003 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2e17bf56]) (logid:27422e16) HA health check task is running... 2025-06-29 04:58:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c00686e6]) (logid:9235b5c6) No inactive management server node found 2025-06-29 04:58:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c00686e6]) (logid:9235b5c6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:08,861 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f1d2a68]) (logid:0cec03a8) No inactive management server node found 2025-06-29 04:58:08,861 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2f1d2a68]) (logid:0cec03a8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:10,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38e1bcb7]) (logid:c9db70b2) No inactive management server node found 2025-06-29 04:58:10,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-38e1bcb7]) (logid:c9db70b2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:11,004 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e3a0121c]) (logid:d53a5e55) HA health check task is running... 2025-06-29 04:58:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c46dfde]) (logid:749abc51) No inactive management server node found 2025-06-29 04:58:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5c46dfde]) (logid:749abc51) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-168fddbf]) (logid:4d31bf2d) Found 0 routers to update status. 2025-06-29 04:58:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-168fddbf]) (logid:4d31bf2d) Found 0 VPC's to update Redundant State. 2025-06-29 04:58:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-168fddbf]) (logid:4d31bf2d) Found 0 networks to update RvR status. 2025-06-29 04:58:13,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60091433]) (logid:efcd321e) No inactive management server node found 2025-06-29 04:58:13,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-60091433]) (logid:efcd321e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5012f56c]) (logid:c08fe76b) No inactive management server node found 2025-06-29 04:58:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5012f56c]) (logid:c08fe76b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:15,005 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-435e64a9]) (logid:8cf4aec8) HA health check task is running... 2025-06-29 04:58:15,425 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-8845: Seq 7-8845: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:58:15,434 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 7-: Sending 8845 2025-06-29 04:58:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01d313e9]) (logid:104b2c8d) No inactive management server node found 2025-06-29 04:58:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-01d313e9]) (logid:104b2c8d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e30c938]) (logid:abac1af8) No inactive management server node found 2025-06-29 04:58:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e30c938]) (logid:abac1af8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:18,104 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-e0f87aa0]) (logid:4d9243af) AutoScaling Monitor is running... 2025-06-29 04:58:18,188 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-5fbe1c79]) (logid:8b202b3f) Starting to remove all volume_stats rows older than [Sat Jun 28 16:58:18 CEST 2025]. 2025-06-29 04:58:18,188 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-5fbe1c79]) (logid:8b202b3f) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:58:18 CEST 2025]. 2025-06-29 04:58:19,006 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-66238d7c]) (logid:d207d5c4) HA health check task is running... 2025-06-29 04:58:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-344b796d]) (logid:6119cb33) No inactive management server node found 2025-06-29 04:58:19,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-344b796d]) (logid:6119cb33) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:19,498 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-84b9ed7c]) (logid:62204493) Starting to remove all vm_stats rows older than [Sat Jun 28 16:58:19 CEST 2025]. 2025-06-29 04:58:19,533 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-84b9ed7c]) (logid:62204493) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:58:19 CEST 2025]. 2025-06-29 04:58:20,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-705b55f1]) (logid:7fd53a08) No inactive management server node found 2025-06-29 04:58:20,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-705b55f1]) (logid:7fd53a08) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:22,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05285643]) (logid:f740aebc) No inactive management server node found 2025-06-29 04:58:22,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-05285643]) (logid:f740aebc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:22,981 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:58:22,983 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:58:22,985 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) Seq 2-4725683383995203920: Routed from 90520738888109 2025-06-29 04:58:23,007 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-c5d6093a]) (logid:c0eeeb01) HA health check task is running... 2025-06-29 04:58:23,029 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) Seq 1-4725683383995203920: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:58:23,030 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:58:23,031 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) Seq 2-3926857400090362235: Routed from 90520738888109 2025-06-29 04:58:23,074 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-ebe75781]) (logid:e74ab694) Seq 2-3926857400090362235: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:58:23,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9499ff17]) (logid:52fad5e8) No inactive management server node found 2025-06-29 04:58:23,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9499ff17]) (logid:52fad5e8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eca2ef04]) (logid:1438a597) No inactive management server node found 2025-06-29 04:58:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eca2ef04]) (logid:1438a597) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:25,426 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8846: Seq 7-8846: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:58:25,435 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8846 2025-06-29 04:58:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe727666]) (logid:18cd681e) No inactive management server node found 2025-06-29 04:58:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fe727666]) (logid:18cd681e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:27,008 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-34afc082]) (logid:248e00af) HA health check task is running... 2025-06-29 04:58:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bcec7655]) (logid:540f92fa) Enabled non-edge zones available for scan: 1 2025-06-29 04:58:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-bcec7655]) (logid:540f92fa) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:58:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-bcec7655]) (logid:540f92fa) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:58:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-98aaf996]) (logid:0d0a4245) Enabled non-edge zones available for scan: 1 2025-06-29 04:58:28,279 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-98aaf996]) (logid:0d0a4245) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:58:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8083a87]) (logid:c527f63e) No inactive management server node found 2025-06-29 04:58:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a8083a87]) (logid:c527f63e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51492cdb]) (logid:4eef127d) No inactive management server node found 2025-06-29 04:58:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-51492cdb]) (logid:4eef127d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:31,010 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-a21b6076]) (logid:0e90f0e9) HA health check task is running... 2025-06-29 04:58:31,245 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:58:31,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:58:31,250 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) Seq 2-4725683383995203921: Routed from 90520738888109 2025-06-29 04:58:31,304 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) Seq 1-4725683383995203921: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:58:31,342 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:58:31,342 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) Seq 2-3926857400090362236: Routed from 90520738888109 2025-06-29 04:58:31,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d1a3ed6]) (logid:8b574e46) No inactive management server node found 2025-06-29 04:58:31,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d1a3ed6]) (logid:8b574e46) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:31,429 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-ac1d56a5]) (logid:309dda54) Seq 2-3926857400090362236: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:58:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56411546]) (logid:59a9d4f0) No inactive management server node found 2025-06-29 04:58:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-56411546]) (logid:59a9d4f0) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:33,582 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) StorageCollector is running... 2025-06-29 04:58:33,587 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:58:33,587 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:58:33,587 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:58:33,588 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-3938116399158788185: Routed from 90520738888109 2025-06-29 04:58:33,642 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 4-3938116399158788185: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:58:33,645 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:58:33,645 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:58:33,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:58:33,646 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-3926857400090362237: Routed from 90520738888109 2025-06-29 04:58:33,658 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-3926857400090362237: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:58:33,669 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:58:33,669 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:58:33,669 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:58:33,669 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-4725683383995203922: Routed from 90520738888109 2025-06-29 04:58:33,718 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 1-4725683383995203922: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:58:33,728 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:58:33,728 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:58:33,728 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:58:33,729 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-4725683383995203923: Routed from 90520738888109 2025-06-29 04:58:33,812 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 1-4725683383995203923: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:58:33,814 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:58:33,814 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:58:33,814 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:58:33,815 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-3926857400090362238: Routed from 90520738888109 2025-06-29 04:58:33,859 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-acdfab54]) (logid:d5ef3b90) Seq 2-3926857400090362238: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:58:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9346c4e3]) (logid:634c2bad) No inactive management server node found 2025-06-29 04:58:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9346c4e3]) (logid:634c2bad) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:35,011 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-6d592b7d]) (logid:7ea3e591) HA health check task is running... 2025-06-29 04:58:35,426 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8847: Seq 7-8847: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:58:35,436 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8847 2025-06-29 04:58:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e71d29d4]) (logid:509acb87) No inactive management server node found 2025-06-29 04:58:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e71d29d4]) (logid:509acb87) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:37,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fc87c3d]) (logid:0e3183f2) No inactive management server node found 2025-06-29 04:58:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fc87c3d]) (logid:0e3183f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca8ebfae]) (logid:93555442) No inactive management server node found 2025-06-29 04:58:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ca8ebfae]) (logid:93555442) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:39,012 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-7187f672]) (logid:62ebc618) HA health check task is running... 2025-06-29 04:58:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1c0adeb]) (logid:08fb0097) No inactive management server node found 2025-06-29 04:58:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a1c0adeb]) (logid:08fb0097) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc23ba2f]) (logid:fd9f59e5) No inactive management server node found 2025-06-29 04:58:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-cc23ba2f]) (logid:fd9f59e5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:43,013 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-3a42e28d]) (logid:743e2e0b) HA health check task is running... 2025-06-29 04:58:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-efce2075]) (logid:fe03cb97) DbCollector is running... 2025-06-29 04:58:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-32c92587]) (logid:9328be62) Found 0 routers to update status. 2025-06-29 04:58:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-32c92587]) (logid:9328be62) Found 0 VPC's to update Redundant State. 2025-06-29 04:58:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-32c92587]) (logid:9328be62) Found 0 networks to update RvR status. 2025-06-29 04:58:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b547a9a]) (logid:2ad3e1a0) No inactive management server node found 2025-06-29 04:58:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1b547a9a]) (logid:2ad3e1a0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) ManagementServerCollector is running... 2025-06-29 04:58:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) DB properties were already loaded 2025-06-29 04:58:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:58:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Execution of process [4068330] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:58:44,056 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Exit value of process [4068330] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:58:44,056 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Process [4068330] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:58:44,056 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) usage install: null 2025-06-29 04:58:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Sessions active: 0 2025-06-29 04:58:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:58:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068333] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:58:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:58:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068338] for command [/bin/bash -c uname -r ]. 2025-06-29 04:58:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:58:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068339] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:58:44,074 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) system memory from /proc: 4887515136 2025-06-29 04:58:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:58:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068344] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:58:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) free memory from /proc: 348516352 2025-06-29 04:58:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:58:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068349] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:58:44,091 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) used memory from /proc: 990448 2025-06-29 04:58:44,091 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:58:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068350] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:58:44,096 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:58:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068351] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:58:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:58:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068352] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:58:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:58:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068359] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:58:44,119 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:58:44,124 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Successfully executed process [4068360] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail buffersdirect.capacity, 331040 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail buffersdirect.count, 97 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail buffersdirect.used, 331040 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail buffersmapped.capacity, 0 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail buffersmapped.count, 0 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail buffersmapped.used, 0 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail jvmuptime, 4983830 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memoryheap.committed, 671088640 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memoryheap.init, 77594624 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memoryheap.usage, 0.26251570217820913 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorynon-heap.committed, 199360512 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorynon-heap.max, -1 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorynon-heap.usage, -1.96315448E8 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorynon-heap.used, 196315448 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.12067407024793389 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22327602726030593 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.Compressed-Class-Space.usage, 0.012402929365634918 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.Metaspace.usage, 0.9930429641759367 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.PS-Eden-Space.usage, 0.46467636269801416 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorytotal.committed, 870449152 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorytotal.init, 85262336 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorytotal.max, 1908932607 2025-06-29 04:58:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail memorytotal.used, 697440232 2025-06-29 04:58:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail threadsdeadlocks, [] 2025-06-29 04:58:44,128 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail threadsnew.count, 0 2025-06-29 04:58:44,129 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:58:44,131 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-6:[ctx-36d78671]) (logid:2dd8f5fe) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":170756760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4983760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":696644832,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":501124784,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348516352,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.2,0.18],"systemCyclesUsage":[66426590,53845342,4590522736],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:58:44,131 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 165, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":170756760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4983760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":696644832,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":501124784,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348516352,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.2,0.18],"systemCyclesUsage":[66426590,53845342,4590522736],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:58:44,131 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:58:44,136 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-35:[ctx-9f39aeae]) (logid:dc917371) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":170756760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4983760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":696644832,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":501124784,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348516352,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.2,0.18],"systemCyclesUsage":[66426590,53845342,4590522736],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:58:44,136 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 4 ms 2025-06-29 04:58:44,136 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 165, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:58:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":170756760,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":4983760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.08,"totalInit":85262336,"totalUsed":696644832,"totalCommitted":870449152,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":501124784,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":348516352,"systemMemoryUsed":990448,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.08,0.2,0.18],"systemCyclesUsage":[66426590,53845342,4590522736],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:58:44,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13040587]) (logid:8341262f) No inactive management server node found 2025-06-29 04:58:44,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-13040587]) (logid:8341262f) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:45,427 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8848: Seq 7-8848: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:58:45,437 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8848 2025-06-29 04:58:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:58:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-14:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:58:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c74f7385]) (logid:5cde9a30) No inactive management server node found 2025-06-29 04:58:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c74f7385]) (logid:5cde9a30) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:46,643 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:58:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:58:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:58:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-9:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:58:47,014 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-e066d338]) (logid:feb5c418) HA health check task is running... 2025-06-29 04:58:47,105 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:58:47,107 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:58:47,108 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:58:47,110 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:58:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4ed0091]) (logid:bc7c782a) No inactive management server node found 2025-06-29 04:58:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c4ed0091]) (logid:bc7c782a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c2278ef6]) (logid:8c1d346e) VM scheduler.poll is being called at 2025-06-29T02:59:00+0000 2025-06-29 04:58:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c2278ef6]) (logid:8c1d346e) Got 0 scheduled jobs to be executed at 2025-06-29T02:59:00+0000 2025-06-29 04:58:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-c2278ef6]) (logid:8c1d346e) Cleaned up 0 VM scheduled job entries 2025-06-29 04:58:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-ba3180e4]) (logid:dc51c627) VM Operation Thread Running 2025-06-29 04:58:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-05293b87]) (logid:db3b6eef) ClusterDRS.poll is being called at 2025-06-29T02:59:00+0000 2025-06-29 04:58:48,359 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-05293b87]) (logid:db3b6eef) Removed 0 old drs migration plans 2025-06-29 04:58:49,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab908e09]) (logid:ab760751) No inactive management server node found 2025-06-29 04:58:49,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ab908e09]) (logid:ab760751) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:50,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f265cd59]) (logid:d5d1689b) No inactive management server node found 2025-06-29 04:58:50,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f265cd59]) (logid:d5d1689b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:51,015 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-4:[ctx-53882156]) (logid:2d8bb6ae) HA health check task is running... 2025-06-29 04:58:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c9b9952]) (logid:b6b6d25a) No inactive management server node found 2025-06-29 04:58:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7c9b9952]) (logid:b6b6d25a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f808a8e5]) (logid:e49712f2) No inactive management server node found 2025-06-29 04:58:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f808a8e5]) (logid:e49712f2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:55,016 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-86399201]) (logid:fd964a27) HA health check task is running... 2025-06-29 04:58:55,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b7875b0]) (logid:8f0c4173) No inactive management server node found 2025-06-29 04:58:55,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3b7875b0]) (logid:8f0c4173) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:55,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-8850: Seq 7-8850: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:58:55,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) SeqA 7-: Sending 8850 2025-06-29 04:58:56,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d78c206]) (logid:b96339ec) No inactive management server node found 2025-06-29 04:58:56,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d78c206]) (logid:b96339ec) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6b3888b9]) (logid:3dc62043) Enabled non-edge zones available for scan: 1 2025-06-29 04:58:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-6b3888b9]) (logid:3dc62043) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:58:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-6b3888b9]) (logid:3dc62043) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:58:58,274 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7894a6f9]) (logid:2e7968d5) Enabled non-edge zones available for scan: 1 2025-06-29 04:58:58,281 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-7894a6f9]) (logid:2e7968d5) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:58:58,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5685e60e]) (logid:ebe6137d) No inactive management server node found 2025-06-29 04:58:58,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5685e60e]) (logid:ebe6137d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:58:59,017 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-25f52f6e]) (logid:a3f6764e) HA health check task is running... 2025-06-29 04:58:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e76ba8fa]) (logid:c4d01f64) No inactive management server node found 2025-06-29 04:58:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e76ba8fa]) (logid:c4d01f64) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e0387ff]) (logid:b7475a01) No inactive management server node found 2025-06-29 04:59:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8e0387ff]) (logid:b7475a01) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9ab0525]) (logid:4b140761) No inactive management server node found 2025-06-29 04:59:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f9ab0525]) (logid:4b140761) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:03,018 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-d5cd49da]) (logid:c67e0862) HA health check task is running... 2025-06-29 04:59:04,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bd02025]) (logid:e45e4989) No inactive management server node found 2025-06-29 04:59:04,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1bd02025]) (logid:e45e4989) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:05,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-8851: Seq 7-8851: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:59:05,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) SeqA 7-: Sending 8851 2025-06-29 04:59:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5099f7c]) (logid:b1220d1c) No inactive management server node found 2025-06-29 04:59:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5099f7c]) (logid:b1220d1c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:07,020 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-58b2aa97]) (logid:1d170a95) HA health check task is running... 2025-06-29 04:59:07,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2730f17]) (logid:72ce38e1) No inactive management server node found 2025-06-29 04:59:07,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f2730f17]) (logid:72ce38e1) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:08,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dfe4a93]) (logid:4ba49456) No inactive management server node found 2025-06-29 04:59:08,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dfe4a93]) (logid:4ba49456) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:10,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b7b3c47]) (logid:2efca934) No inactive management server node found 2025-06-29 04:59:10,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8b7b3c47]) (logid:2efca934) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:11,021 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e13331a3]) (logid:424fca18) HA health check task is running... 2025-06-29 04:59:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f45e94e]) (logid:710d2823) No inactive management server node found 2025-06-29 04:59:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9f45e94e]) (logid:710d2823) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1f687a90]) (logid:2e0e5793) Found 0 routers to update status. 2025-06-29 04:59:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1f687a90]) (logid:2e0e5793) Found 0 VPC's to update Redundant State. 2025-06-29 04:59:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1f687a90]) (logid:2e0e5793) Found 0 networks to update RvR status. 2025-06-29 04:59:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c866f8cf]) (logid:e7d97b92) No inactive management server node found 2025-06-29 04:59:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c866f8cf]) (logid:e7d97b92) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f210e06]) (logid:16c8f1d1) No inactive management server node found 2025-06-29 04:59:14,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6f210e06]) (logid:16c8f1d1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:15,022 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e769e602]) (logid:1aa2df24) HA health check task is running... 2025-06-29 04:59:15,428 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-8852: Seq 7-8852: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:59:15,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) SeqA 7-: Sending 8852 2025-06-29 04:59:16,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e2898b6]) (logid:260723c8) No inactive management server node found 2025-06-29 04:59:16,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3e2898b6]) (logid:260723c8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd1b6865]) (logid:d3c98d31) No inactive management server node found 2025-06-29 04:59:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd1b6865]) (logid:d3c98d31) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:18,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-95090688]) (logid:176967ed) Resetting hosts suitable for reconnect 2025-06-29 04:59:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-95090688]) (logid:176967ed) Completed resetting hosts suitable for reconnect 2025-06-29 04:59:18,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-95090688]) (logid:176967ed) Acquiring hosts for clusters already owned by this management server 2025-06-29 04:59:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-95090688]) (logid:176967ed) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 04:59:18,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-95090688]) (logid:176967ed) Acquiring hosts for clusters not owned by any management server 2025-06-29 04:59:18,099 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-95090688]) (logid:176967ed) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 04:59:18,106 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-2:[ctx-82199e52]) (logid:ed34fa2c) AutoScaling Monitor is running... 2025-06-29 04:59:18,191 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-ea66fb3f]) (logid:2181fd97) Starting to remove all volume_stats rows older than [Sat Jun 28 16:59:18 CEST 2025]. 2025-06-29 04:59:18,191 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:[ctx-ea66fb3f]) (logid:2181fd97) Removed a total of [0] volume_stats rows older than [Sat Jun 28 16:59:18 CEST 2025]. 2025-06-29 04:59:19,023 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-376f9480]) (logid:d3b9537e) HA health check task is running... 2025-06-29 04:59:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-609dd06d]) (logid:97c8dbd1) No inactive management server node found 2025-06-29 04:59:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-609dd06d]) (logid:97c8dbd1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:19,534 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-90741e43]) (logid:25567a60) Starting to remove all vm_stats rows older than [Sat Jun 28 16:59:19 CEST 2025]. 2025-06-29 04:59:19,558 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-90741e43]) (logid:25567a60) Removed a total of [8] vm_stats rows older than [Sat Jun 28 16:59:19 CEST 2025]. 2025-06-29 04:59:20,428 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8853: Seq 7-8853: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:59:20,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8853 2025-06-29 04:59:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-329e49ba]) (logid:fa29b201) No inactive management server node found 2025-06-29 04:59:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-329e49ba]) (logid:fa29b201) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3073b822]) (logid:871dad44) No inactive management server node found 2025-06-29 04:59:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3073b822]) (logid:871dad44) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:23,024 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e4c1e53c]) (logid:a78d61e0) HA health check task is running... 2025-06-29 04:59:23,077 DEBUG [c.c.s.S.HostCollector] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) HostStatsCollector is running to process 2 UP hosts 2025-06-29 04:59:23,079 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:59:23,080 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) Seq 2-4725683383995203924: Routed from 90520738888109 2025-06-29 04:59:23,123 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) Seq 1-4725683383995203924: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:59:23,125 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 04:59:23,125 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) Seq 2-3926857400090362239: Routed from 90520738888109 2025-06-29 04:59:23,169 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-7b1a44fa]) (logid:228ee40c) Seq 2-3926857400090362239: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 04:59:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5be6a701]) (logid:69b7a503) No inactive management server node found 2025-06-29 04:59:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5be6a701]) (logid:69b7a503) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:25,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f27dfdf0]) (logid:df1b2f8f) No inactive management server node found 2025-06-29 04:59:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f27dfdf0]) (logid:df1b2f8f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15494ac2]) (logid:30df3343) No inactive management server node found 2025-06-29 04:59:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-15494ac2]) (logid:30df3343) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:27,025 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-618c06f8]) (logid:b7c4053d) HA health check task is running... 2025-06-29 04:59:28,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-83ffe565]) (logid:c2b39037) Enabled non-edge zones available for scan: 1 2025-06-29 04:59:28,096 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-83ffe565]) (logid:c2b39037) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:59:28,097 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-83ffe565]) (logid:c2b39037) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:59:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a491b02f]) (logid:1aff82a6) Enabled non-edge zones available for scan: 1 2025-06-29 04:59:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-a491b02f]) (logid:1aff82a6) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:59:28,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fd3c841]) (logid:7742aa42) No inactive management server node found 2025-06-29 04:59:28,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0fd3c841]) (logid:7742aa42) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:29,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9119a07e]) (logid:f2a7f72c) No inactive management server node found 2025-06-29 04:59:29,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9119a07e]) (logid:f2a7f72c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:30,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8854: Seq 7-8854: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:59:30,440 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8854 2025-06-29 04:59:31,026 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-91ced1bd]) (logid:db3c40db) HA health check task is running... 2025-06-29 04:59:31,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a140e636]) (logid:e0fc1ac7) No inactive management server node found 2025-06-29 04:59:31,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a140e636]) (logid:e0fc1ac7) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:31,483 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 04:59:31,488 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:59:31,489 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) Seq 2-4725683383995203925: Routed from 90520738888109 2025-06-29 04:59:31,544 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) Seq 1-4725683383995203925: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:59:31,581 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 04:59:31,582 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) Seq 2-3926857400090362240: Routed from 90520738888109 2025-06-29 04:59:31,667 DEBUG [c.c.a.t.Request] (StatsCollector-1:[ctx-4c52b6da]) (logid:59296abd) Seq 2-3926857400090362240: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 04:59:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e43fea3]) (logid:cd104e03) No inactive management server node found 2025-06-29 04:59:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0e43fea3]) (logid:cd104e03) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:33,859 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) StorageCollector is running... 2025-06-29 04:59:33,866 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:59:33,866 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:59:33,866 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:59:33,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 2-3938116399158788186: Routed from 90520738888109 2025-06-29 04:59:33,920 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 4-3938116399158788186: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:59:33,923 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:59:33,923 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:59:33,923 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:59:33,924 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 2-4725683383995203926: Routed from 90520738888109 2025-06-29 04:59:33,971 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 1-4725683383995203926: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:59:33,982 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:59:33,982 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:59:33,982 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:59:33,983 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 2-4725683383995203927: Routed from 90520738888109 2025-06-29 04:59:34,073 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 1-4725683383995203927: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:59:34,084 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:59:34,084 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:59:34,084 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:59:34,085 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 2-4725683383995203928: Routed from 90520738888109 2025-06-29 04:59:34,171 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 1-4725683383995203928: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:59:34,173 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 04:59:34,173 DEBUG [c.c.h.XenServerGuru] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 04:59:34,173 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 04:59:34,174 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 2-3926857400090362241: Routed from 90520738888109 2025-06-29 04:59:34,177 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-47dcf27d]) (logid:ebd83b36) Seq 2-3926857400090362241: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 04:59:34,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dfd466a]) (logid:e5d7081c) No inactive management server node found 2025-06-29 04:59:34,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dfd466a]) (logid:e5d7081c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:35,027 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2d2e8fc1]) (logid:84f972d5) HA health check task is running... 2025-06-29 04:59:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5c2febd]) (logid:8fba1db2) No inactive management server node found 2025-06-29 04:59:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b5c2febd]) (logid:8fba1db2) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:37,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-827b98d8]) (logid:e39c8579) No inactive management server node found 2025-06-29 04:59:37,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-827b98d8]) (logid:e39c8579) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57faaeb9]) (logid:81d3a3b1) No inactive management server node found 2025-06-29 04:59:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-57faaeb9]) (logid:81d3a3b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:39,028 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-ac025bdf]) (logid:7feea1c6) HA health check task is running... 2025-06-29 04:59:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-795ff64b]) (logid:9ccb45f2) No inactive management server node found 2025-06-29 04:59:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-795ff64b]) (logid:9ccb45f2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:40,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8855: Seq 7-8855: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:59:40,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8855 2025-06-29 04:59:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55592e68]) (logid:85139401) No inactive management server node found 2025-06-29 04:59:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-55592e68]) (logid:85139401) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:43,029 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-9063946c]) (logid:a3e725a1) HA health check task is running... 2025-06-29 04:59:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-2:[ctx-9d37fab2]) (logid:92aa69dc) DbCollector is running... 2025-06-29 04:59:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1e5928b5]) (logid:e779accf) Found 0 routers to update status. 2025-06-29 04:59:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1e5928b5]) (logid:e779accf) Found 0 VPC's to update Redundant State. 2025-06-29 04:59:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-1e5928b5]) (logid:e779accf) Found 0 networks to update RvR status. 2025-06-29 04:59:43,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68ad8adf]) (logid:33539e90) No inactive management server node found 2025-06-29 04:59:43,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-68ad8adf]) (logid:33539e90) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) ManagementServerCollector is running... 2025-06-29 04:59:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) DB properties were already loaded 2025-06-29 04:59:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 04:59:44,057 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Execution of process [4068371] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 04:59:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Exit value of process [4068371] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 04:59:44,058 WARN [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Process [4068371] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 04:59:44,059 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) usage install: null 2025-06-29 04:59:44,059 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Sessions active: 0 2025-06-29 04:59:44,061 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:59:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068374] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 04:59:44,070 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c uname -r ]. 2025-06-29 04:59:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068379] for command [/bin/bash -c uname -r ]. 2025-06-29 04:59:44,074 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:59:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068380] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:59:44,080 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) system memory from /proc: 4887515136 2025-06-29 04:59:44,080 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:59:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068385] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 04:59:44,086 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) free memory from /proc: 349442048 2025-06-29 04:59:44,086 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:59:44,098 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068390] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 04:59:44,098 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) used memory from /proc: 990576 2025-06-29 04:59:44,099 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 04:59:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068391] for command [/bin/bash -c uptime -s ]. 2025-06-29 04:59:44,103 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:59:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068392] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 04:59:44,115 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:59:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068393] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 04:59:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:59:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068400] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 04:59:44,126 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:59:44,131 DEBUG [c.c.u.s.Script] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Successfully executed process [4068401] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail buffersdirect.capacity, 331040 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail buffersdirect.count, 97 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail buffersdirect.used, 331040 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail buffersmapped.capacity, 0 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail buffersmapped.count, 0 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail buffersmapped.used, 0 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail gcPS-Scavenge.count, 110 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail gcPS-Scavenge.time, 582 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail jvmuptime, 5043836 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memoryheap.committed, 671088640 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memoryheap.init, 77594624 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memoryheap.usage, 0.27440766730304605 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorynon-heap.committed, 199426048 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorynon-heap.init, 7667712 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorynon-heap.max, -1 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorynon-heap.usage, -1.96435072E8 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorynon-heap.used, 196435072 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.12131035890429219 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2235145132802346 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.Compressed-Class-Space.usage, 0.012402929365634918 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.Metaspace.usage, 0.9931294711468092 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.PS-Eden-Space.usage, 0.499907156041815 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.PS-Old-Gen.usage, 0.12614906336177797 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorypools.PS-Survivor-Space.usage, 0.9809592642435213 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorytotal.committed, 870514688 2025-06-29 04:59:44,133 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorytotal.init, 85262336 2025-06-29 04:59:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorytotal.max, 1908932607 2025-06-29 04:59:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail memorytotal.used, 720260816 2025-06-29 04:59:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail threadsdeadlocks, [] 2025-06-29 04:59:44,134 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail threadsnew.count, 0 2025-06-29 04:59:44,135 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) not storing detail threadstimed_waiting.count, 84 2025-06-29 04:59:44,137 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:[ctx-32f90045]) (logid:be21871b) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":148109456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5043762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":719399536,"totalCommitted":870514688,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":523825744,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":349442048,"systemMemoryUsed":990576,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.18,0.18],"systemCyclesUsage":[66426929,53845641,4590545856],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 04:59:44,137 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 167, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":148109456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5043762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":719399536,"totalCommitted":870514688,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":523825744,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":349442048,"systemMemoryUsed":990576,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.18,0.18],"systemCyclesUsage":[66426929,53845641,4590545856],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:59:44,137 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 04:59:44,143 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 04:59:44,143 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-3:[ctx-1f9919b2]) (logid:47940bd5) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 167, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":148109456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5043762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":719399536,"totalCommitted":870514688,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":523825744,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":349442048,"systemMemoryUsed":990576,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.18,0.18],"systemCyclesUsage":[66426929,53845641,4590545856],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:59:44,143 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-36:[ctx-8309d032]) (logid:f8382daf) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 4:59:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":671088640,"freeJvmMemoryBytes":148109456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5043762,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.1,"totalInit":85262336,"totalUsed":719399536,"totalCommitted":870514688,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":523825744,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":349442048,"systemMemoryUsed":990576,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.1,0.18,0.18],"systemCyclesUsage":[66426929,53845641,4590545856],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 04:59:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71455a90]) (logid:cd6f5aef) No inactive management server node found 2025-06-29 04:59:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-71455a90]) (logid:cd6f5aef) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 04:59:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-5:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 04:59:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad1f4daf]) (logid:efefada0) No inactive management server node found 2025-06-29 04:59:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ad1f4daf]) (logid:efefada0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:46,644 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-2:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:59:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:59:46,646 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 04:59:46,648 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 04:59:47,031 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-e1220bd6]) (logid:81c82d1d) HA health check task is running... 2025-06-29 04:59:47,068 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:59:47,069 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:59:47,070 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 04:59:47,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 04:59:47,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e8548439]) (logid:3aa27db7) No inactive management server node found 2025-06-29 04:59:47,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e8548439]) (logid:3aa27db7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f8622f13]) (logid:c1ec33e4) VM scheduler.poll is being called at 2025-06-29T03:00:00+0000 2025-06-29 04:59:48,159 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f8622f13]) (logid:c1ec33e4) Got 0 scheduled jobs to be executed at 2025-06-29T03:00:00+0000 2025-06-29 04:59:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-f8622f13]) (logid:c1ec33e4) Cleaned up 0 VM scheduled job entries 2025-06-29 04:59:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-d9338339]) (logid:bc014c48) VM Operation Thread Running 2025-06-29 04:59:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f455e33d]) (logid:e1d6d184) ClusterDRS.poll is being called at 2025-06-29T03:00:00+0000 2025-06-29 04:59:48,358 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-f455e33d]) (logid:e1d6d184) Removed 0 old drs migration plans 2025-06-29 04:59:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f645247e]) (logid:7b6b8094) No inactive management server node found 2025-06-29 04:59:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f645247e]) (logid:7b6b8094) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:50,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-8857: Seq 7-8857: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 04:59:50,440 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) SeqA 7-: Sending 8857 2025-06-29 04:59:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49491818]) (logid:1390af1e) No inactive management server node found 2025-06-29 04:59:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-49491818]) (logid:1390af1e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:51,032 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b1fa740b]) (logid:b6459511) HA health check task is running... 2025-06-29 04:59:52,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-855400f6]) (logid:2a99b2ed) No inactive management server node found 2025-06-29 04:59:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-855400f6]) (logid:2a99b2ed) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:53,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5bee740]) (logid:b3e54469) No inactive management server node found 2025-06-29 04:59:53,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d5bee740]) (logid:b3e54469) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:55,033 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7767f20d]) (logid:1af94df0) HA health check task is running... 2025-06-29 04:59:55,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d48f826]) (logid:575aba62) No inactive management server node found 2025-06-29 04:59:55,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-3d48f826]) (logid:575aba62) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:56,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b17309b]) (logid:0e1f7dff) No inactive management server node found 2025-06-29 04:59:56,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b17309b]) (logid:0e1f7dff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 04:59:58,089 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d85b1068]) (logid:d03c600f) Enabled non-edge zones available for scan: 1 2025-06-29 04:59:58,094 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-d85b1068]) (logid:d03c600f) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 04:59:58,095 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-d85b1068]) (logid:d03c600f) Zone [1] is ready to launch secondary storage VM. 2025-06-29 04:59:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d7eea6e0]) (logid:8bd08cfd) Enabled non-edge zones available for scan: 1 2025-06-29 04:59:58,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-d7eea6e0]) (logid:8bd08cfd) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 04:59:58,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39630b98]) (logid:8ce0feb8) No inactive management server node found 2025-06-29 04:59:58,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39630b98]) (logid:8ce0feb8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 04:59:59,034 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7f6150a5]) (logid:b57d7c6b) HA health check task is running... 2025-06-29 04:59:59,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed1a9053]) (logid:ef70c215) No inactive management server node found 2025-06-29 04:59:59,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ed1a9053]) (logid:ef70c215) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:00,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-8858: Seq 7-8858: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:00:00,440 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) SeqA 7-: Sending 8858 2025-06-29 05:00:01,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76690868]) (logid:c72fa3bf) No inactive management server node found 2025-06-29 05:00:01,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-76690868]) (logid:c72fa3bf) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a3da93d]) (logid:47f2afdb) No inactive management server node found 2025-06-29 05:00:02,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a3da93d]) (logid:47f2afdb) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:03,035 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-5adb186d]) (logid:f67d1112) HA health check task is running... 2025-06-29 05:00:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-891afc74]) (logid:d14b477f) No inactive management server node found 2025-06-29 05:00:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-891afc74]) (logid:d14b477f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:05,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b592cb1f]) (logid:6acc93ff) No inactive management server node found 2025-06-29 05:00:05,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b592cb1f]) (logid:6acc93ff) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:07,036 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-02f6f069]) (logid:c1132fa4) HA health check task is running... 2025-06-29 05:00:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1889798a]) (logid:2a832b18) No inactive management server node found 2025-06-29 05:00:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1889798a]) (logid:2a832b18) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:08,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8644cc11]) (logid:59e0a832) No inactive management server node found 2025-06-29 05:00:08,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8644cc11]) (logid:59e0a832) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:10,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bffc3ace]) (logid:213bf717) No inactive management server node found 2025-06-29 05:00:10,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bffc3ace]) (logid:213bf717) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:10,430 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-8859: Seq 7-8859: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:00:10,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) SeqA 7-: Sending 8859 2025-06-29 05:00:11,038 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-b57a3892]) (logid:429522eb) HA health check task is running... 2025-06-29 05:00:11,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45a0f05a]) (logid:b11003b1) No inactive management server node found 2025-06-29 05:00:11,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45a0f05a]) (logid:b11003b1) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7d095ba4]) (logid:c204de69) Found 0 routers to update status. 2025-06-29 05:00:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7d095ba4]) (logid:c204de69) Found 0 VPC's to update Redundant State. 2025-06-29 05:00:13,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-7d095ba4]) (logid:c204de69) Found 0 networks to update RvR status. 2025-06-29 05:00:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-853c6a99]) (logid:aafdcaa5) No inactive management server node found 2025-06-29 05:00:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-853c6a99]) (logid:aafdcaa5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:14,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d305e77]) (logid:7d9afc90) No inactive management server node found 2025-06-29 05:00:14,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9d305e77]) (logid:7d9afc90) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:15,039 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-03e5b98a]) (logid:e638f52d) HA health check task is running... 2025-06-29 05:00:16,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e71c2f81]) (logid:2b00f104) No inactive management server node found 2025-06-29 05:00:16,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e71c2f81]) (logid:2b00f104) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:17,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb10009a]) (logid:ee01bbb8) No inactive management server node found 2025-06-29 05:00:17,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb10009a]) (logid:ee01bbb8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:18,108 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-5:[ctx-cf0d536d]) (logid:895ff477) AutoScaling Monitor is running... 2025-06-29 05:00:18,193 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-3fddd9e5]) (logid:8b044aff) Starting to remove all volume_stats rows older than [Sat Jun 28 17:00:18 CEST 2025]. 2025-06-29 05:00:18,194 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-4:[ctx-3fddd9e5]) (logid:8b044aff) Removed a total of [0] volume_stats rows older than [Sat Jun 28 17:00:18 CEST 2025]. 2025-06-29 05:00:19,040 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-844b1d1a]) (logid:7c609267) HA health check task is running... 2025-06-29 05:00:19,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65d1b955]) (logid:3d1cad70) No inactive management server node found 2025-06-29 05:00:19,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-65d1b955]) (logid:3d1cad70) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:19,559 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-b88c5b66]) (logid:f847dbc8) Starting to remove all vm_stats rows older than [Sat Jun 28 17:00:19 CEST 2025]. 2025-06-29 05:00:19,583 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-1:[ctx-b88c5b66]) (logid:f847dbc8) Removed a total of [8] vm_stats rows older than [Sat Jun 28 17:00:19 CEST 2025]. 2025-06-29 05:00:20,431 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8860: Seq 7-8860: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:00:20,441 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8860 2025-06-29 05:00:20,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9d6515f]) (logid:87112f28) No inactive management server node found 2025-06-29 05:00:20,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d9d6515f]) (logid:87112f28) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba54145c]) (logid:7bd7c7b7) No inactive management server node found 2025-06-29 05:00:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ba54145c]) (logid:7bd7c7b7) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:23,041 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-0b12d0da]) (logid:fad47272) HA health check task is running... 2025-06-29 05:00:23,172 DEBUG [c.c.s.S.HostCollector] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) HostStatsCollector is running to process 2 UP hosts 2025-06-29 05:00:23,174 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 05:00:23,175 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) Seq 2-4725683383995203929: Routed from 90520738888109 2025-06-29 05:00:23,218 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) Seq 1-4725683383995203929: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 05:00:23,220 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 05:00:23,221 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) Seq 2-3926857400090362242: Routed from 90520738888109 2025-06-29 05:00:23,264 DEBUG [c.c.a.t.Request] (StatsCollector-6:[ctx-25306f56]) (logid:0ef86fda) Seq 2-3926857400090362242: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 05:00:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-838dde50]) (logid:59465b70) No inactive management server node found 2025-06-29 05:00:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-838dde50]) (logid:59465b70) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:25,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-737d1649]) (logid:7e379b77) No inactive management server node found 2025-06-29 05:00:25,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-737d1649]) (logid:7e379b77) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39b848e0]) (logid:1f73b32c) No inactive management server node found 2025-06-29 05:00:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-39b848e0]) (logid:1f73b32c) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:27,042 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-944e2460]) (logid:ea5f46b9) HA health check task is running... 2025-06-29 05:00:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-13a95002]) (logid:3ed110b1) Enabled non-edge zones available for scan: 1 2025-06-29 05:00:28,093 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-13a95002]) (logid:3ed110b1) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 05:00:28,094 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-13a95002]) (logid:3ed110b1) Zone [1] is ready to launch secondary storage VM. 2025-06-29 05:00:28,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f5cbfbf9]) (logid:3bc39d45) Enabled non-edge zones available for scan: 1 2025-06-29 05:00:28,277 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-f5cbfbf9]) (logid:3bc39d45) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 05:00:28,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e44193b1]) (logid:6339e672) No inactive management server node found 2025-06-29 05:00:28,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e44193b1]) (logid:6339e672) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:29,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e9aebc4]) (logid:52f9c3b5) No inactive management server node found 2025-06-29 05:00:29,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4e9aebc4]) (logid:52f9c3b5) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:30,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8861: Seq 7-8861: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:00:30,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8861 2025-06-29 05:00:31,043 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-80663be1]) (logid:85edf933) HA health check task is running... 2025-06-29 05:00:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c93a4fa]) (logid:c8f2628f) No inactive management server node found 2025-06-29 05:00:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6c93a4fa]) (logid:c8f2628f) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:31,720 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 05:00:31,723 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 05:00:31,724 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) Seq 2-4725683383995203930: Routed from 90520738888109 2025-06-29 05:00:31,778 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) Seq 1-4725683383995203930: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 05:00:31,812 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 05:00:31,813 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) Seq 2-3926857400090362243: Routed from 90520738888109 2025-06-29 05:00:31,899 DEBUG [c.c.a.t.Request] (StatsCollector-2:[ctx-653a0b4b]) (logid:05389239) Seq 2-3926857400090362243: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 05:00:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a91e3606]) (logid:5b6c9625) No inactive management server node found 2025-06-29 05:00:32,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-a91e3606]) (logid:5b6c9625) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:34,177 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) StorageCollector is running... 2025-06-29 05:00:34,184 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:00:34,184 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:00:34,184 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:00:34,185 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-3938116399158788187: Routed from 90520738888109 2025-06-29 05:00:34,239 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 4-3938116399158788187: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:00:34,244 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:00:34,244 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:00:34,245 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:00:34,246 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-4725683383995203931: Routed from 90520738888109 2025-06-29 05:00:34,294 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 1-4725683383995203931: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:00:34,306 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:00:34,306 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:00:34,306 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:00:34,308 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-3926857400090362244: Routed from 90520738888109 2025-06-29 05:00:34,320 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-3926857400090362244: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:00:34,331 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:00:34,331 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:00:34,331 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:00:34,332 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-4725683383995203932: Routed from 90520738888109 2025-06-29 05:00:34,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ff4b2b1]) (logid:1c092df8) No inactive management server node found 2025-06-29 05:00:34,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6ff4b2b1]) (logid:1c092df8) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:34,375 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 1-4725683383995203932: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:00:34,377 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:00:34,377 DEBUG [c.c.h.XenServerGuru] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:00:34,377 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:00:34,378 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-3926857400090362245: Routed from 90520738888109 2025-06-29 05:00:34,422 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-9418e4b5]) (logid:686d6b88) Seq 2-3926857400090362245: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:00:35,044 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-dc4715b1]) (logid:bc9a9780) HA health check task is running... 2025-06-29 05:00:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a71c3cb]) (logid:a9a449e5) No inactive management server node found 2025-06-29 05:00:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2a71c3cb]) (logid:a9a449e5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-abcec977]) (logid:ecceb6a0) No inactive management server node found 2025-06-29 05:00:37,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-abcec977]) (logid:ecceb6a0) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:38,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98549b30]) (logid:9550e1f3) No inactive management server node found 2025-06-29 05:00:38,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-98549b30]) (logid:9550e1f3) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:39,046 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-2cf1df9d]) (logid:1e50b73d) HA health check task is running... 2025-06-29 05:00:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcc69e1f]) (logid:488b14c6) No inactive management server node found 2025-06-29 05:00:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-dcc69e1f]) (logid:488b14c6) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:40,432 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-8862: Seq 7-8862: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:00:40,446 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 05:00:40,446 ERROR [c.c.u.d.T.Transaction] (Finalizer:[]) (logid:) Something went wrong that a transaction is orphaned before db connection is closed 2025-06-29 05:00:40,455 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-12:[]) (logid:) SeqA 7-: Sending 8862 2025-06-29 05:00:41,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-caffd090]) (logid:c6928641) No inactive management server node found 2025-06-29 05:00:41,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-caffd090]) (logid:c6928641) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-5:[ctx-4618f9de]) (logid:89a72e07) DbCollector is running... 2025-06-29 05:00:43,047 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-1:[ctx-7526f206]) (logid:477150a2) HA health check task is running... 2025-06-29 05:00:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-58af2319]) (logid:71ca93fc) Found 0 routers to update status. 2025-06-29 05:00:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-58af2319]) (logid:71ca93fc) Found 0 VPC's to update Redundant State. 2025-06-29 05:00:43,101 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-58af2319]) (logid:71ca93fc) Found 0 networks to update RvR status. 2025-06-29 05:00:43,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dc205ae]) (logid:6428c1f5) No inactive management server node found 2025-06-29 05:00:43,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9dc205ae]) (logid:6428c1f5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) ManagementServerCollector is running... 2025-06-29 05:00:44,041 DEBUG [c.c.u.d.DbProperties] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) DB properties were already loaded 2025-06-29 05:00:44,041 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 05:00:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Execution of process [4068409] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 05:00:44,057 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Exit value of process [4068409] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 05:00:44,057 WARN [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Process [4068409] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 05:00:44,057 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) usage install: null 2025-06-29 05:00:44,057 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Sessions active: 0 2025-06-29 05:00:44,059 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 05:00:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068412] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 05:00:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c uname -r ]. 2025-06-29 05:00:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068421] for command [/bin/bash -c uname -r ]. 2025-06-29 05:00:44,071 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:00:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068424] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:00:44,077 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) system memory from /proc: 4887515136 2025-06-29 05:00:44,077 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:00:44,082 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068429] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:00:44,083 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) free memory from /proc: 363311104 2025-06-29 05:00:44,083 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 05:00:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068434] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 05:00:44,095 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) used memory from /proc: 984480 2025-06-29 05:00:44,095 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 05:00:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068435] for command [/bin/bash -c uptime -s ]. 2025-06-29 05:00:44,100 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 05:00:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068436] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 05:00:44,111 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 05:00:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068437] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 05:00:44,118 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 05:00:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068444] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 05:00:44,122 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 05:00:44,128 DEBUG [c.c.u.s.Script] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Successfully executed process [4068445] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail buffersdirect.capacity, 331040 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail buffersdirect.count, 97 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail buffersdirect.used, 331040 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail buffersmapped.capacity, 0 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail buffersmapped.count, 0 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail buffersmapped.used, 0 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail gcPS-MarkSweep.count, 6 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail gcPS-MarkSweep.time, 993 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail gcPS-Scavenge.count, 111 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail gcPS-Scavenge.time, 595 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail jvmname, 4064514@cs-mgmt1.cloud.redacted.net 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail jvmuptime, 5103833 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail jvmvendor, Red Hat, Inc. OpenJDK 64-Bit Server VM 17.0.15+6-LTS (17) 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memoryheap.committed, 675282944 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memoryheap.init, 77594624 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memoryheap.usage, 0.10781043140942564 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorynon-heap.committed, 199491584 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorynon-heap.init, 7667712 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorynon-heap.max, -1 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorynon-heap.usage, -1.96479536E8 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorynon-heap.used, 196479536 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3044517543859649 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.12159465809117569 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.22357387442929982 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.Compressed-Class-Space.usage, 0.012402929365634918 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.Metaspace.usage, 0.9931453660165398 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.PS-Eden-Space.usage, 0.009911303947393311 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.PS-Old-Gen.usage, 0.12615478470926972 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorypools.PS-Survivor-Space.usage, 0.9763645481418919 2025-06-29 05:00:44,130 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorytotal.committed, 874774528 2025-06-29 05:00:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorytotal.init, 85262336 2025-06-29 05:00:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorytotal.max, 1908932607 2025-06-29 05:00:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail memorytotal.used, 402490280 2025-06-29 05:00:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail threadsdeadlocks, [] 2025-06-29 05:00:44,131 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail threadsnew.count, 0 2025-06-29 05:00:44,132 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) not storing detail threadstimed_waiting.count, 84 2025-06-29 05:00:44,133 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-4:[ctx-4e813fe1]) (logid:e4891f96) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 5:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":675282944,"freeJvmMemoryBytes":471435152,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":400327328,"totalCommitted":874774528,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205802848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":363311104,"systemMemoryUsed":984480,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.18,0.18],"systemCyclesUsage":[66427249,53845933,4590569004],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} to ManagementServer {"id":1,"msid":90520738888109,"name":"cs-mgmt1.cloud.redacted.net","uuid":"3a04c60c-a438-4834-b764-71905f8680f8"} 2025-06-29 05:00:44,134 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109. agent: 0, pdu seq: 169, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 5:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":675282944,"freeJvmMemoryBytes":471435152,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":400327328,"totalCommitted":874774528,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205802848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":363311104,"systemMemoryUsed":984480,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.18,0.18],"systemCyclesUsage":[66427249,53845933,4590569004],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 05:00:44,134 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Executing ClusterServicePdu with service URL: https://172.16.20.40:9090/clusterservice 2025-06-29 05:00:44,139 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-36:[ctx-087d920a]) (logid:03932c41) StatusUpdate from 90520738888109, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 5:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":675282944,"freeJvmMemoryBytes":471435152,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":400327328,"totalCommitted":874774528,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205802848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":363311104,"systemMemoryUsed":984480,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.18,0.18],"systemCyclesUsage":[66427249,53845933,4590569004],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 05:00:44,140 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) POST https://172.16.20.40:9090/clusterservice response :true, responding time: 5 ms 2025-06-29 05:00:44,140 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-1:[ctx-00b10420]) (logid:982a4694) Cluster PDU 90520738888109 -> 90520738888109 completed. time: 5ms. agent: 0, pdu seq: 169, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"3a04c60c-a438-4834-b764-71905f8680f8","managementServerRunId":1751160968845,"collectionTime":"Jun 29, 2025, 5:00:44 AM","sessions":0,"cpuUtilization":0.0,"totalJvmMemoryBytes":675282944,"freeJvmMemoryBytes":471435152,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":5103760,"jvmStartTime":1751160940347,"availableProcessors":4,"loadAverage":0.15,"totalInit":85262336,"totalUsed":400327328,"totalCommitted":874774528,"pid":4064514,"jvmName":"4064514@cs-mgmt1.cloud.redacted.net","jvmVendor":"Red Hat, Inc.","jvmVersion":"17.0.15+6-LTS","osDistribution":"Red Hat Enterprise Linux 9.6 (Plow)","lastAgents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agents":["ddcf716a-cb61-4697-8501-9320c3f30a4b","dfd0102e-f60c-44b4-8a12-70bc46fc8154","cc5764b8-77a5-4aa3-8141-f1334e493361","fc16264e-ee6a-403f-809d-20f1822cb15a"],"agentCount":4,"heapMemoryUsed":205802848,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":16,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":204,"threadsWaitingCount":111,"systemMemoryTotal":4887515136,"systemMemoryFree":363311104,"systemMemoryUsed":984480,"systemMemoryVirtualSize":6220464128,"logInfo":"","systemTotalCpuCycles":9578.24,"systemLoadAverages":[0.15,0.18,0.18],"systemCyclesUsage":[66427249,53845933,4590569004],"dbLocal":true,"usageLocal":false,"systemBootTime":"Feb 11, 2025, 7:33:19 AM","kernelVersion":"5.14.0-427.33.1.el9_4.x86_64"} 2025-06-29 05:00:44,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a2d303d]) (logid:4c8cd61e) No inactive management server node found 2025-06-29 05:00:44,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5a2d303d]) (logid:4c8cd61e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:45,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Ping from AgentAttache {"_id":4,"_name":"s-2-VM","_uuid":"cc5764b8-77a5-4aa3-8141-f1334e493361"} 2025-06-29 05:00:45,924 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-13:[]) (logid:) Ping from AgentAttache {"_id":7,"_name":"v-306-VM","_uuid":"fc16264e-ee6a-403f-809d-20f1822cb15a"} 2025-06-29 05:00:46,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0572f67]) (logid:e9c58bc9) No inactive management server node found 2025-06-29 05:00:46,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b0572f67]) (logid:e9c58bc9) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:46,642 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-1:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"cs-compute1.cloud.redacted.net","_uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 05:00:46,644 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 05:00:46,645 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Process VM state report. Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"}, number of records in report: 3. VMs: [50:PowerOn, 74:PowerOn, 303:PowerOn]] 2025-06-29 05:00:46,647 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"cs-compute1.cloud.redacted.net","type":"Routing","uuid":"ddcf716a-cb61-4697-8501-9320c3f30a4b"} 2025-06-29 05:00:47,048 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-2:[ctx-8e75fe90]) (logid:d2d76fa1) HA health check task is running... 2025-06-29 05:00:47,093 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) Ping from Routing host AgentAttache {"_id":2,"_name":"cs-compute2.cloud.redacted.net","_uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 05:00:47,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process host VM state report from ping process. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 05:00:47,097 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Process VM state report. Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"}, number of records in report: 5. VMs: [2:PowerOn, 306:PowerOn, 110:PowerOn, 15:PowerOn, 127:PowerOn]] 2025-06-29 05:00:47,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:[]) (logid:) Done with process of VM state report. host: Host {"id":2,"name":"cs-compute2.cloud.redacted.net","type":"Routing","uuid":"dfd0102e-f60c-44b4-8a12-70bc46fc8154"} 2025-06-29 05:00:47,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7716bcd2]) (logid:33a4ca76) No inactive management server node found 2025-06-29 05:00:47,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-7716bcd2]) (logid:33a4ca76) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:48,096 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-94b15a01]) (logid:8b982703) Resetting hosts suitable for reconnect 2025-06-29 05:00:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-94b15a01]) (logid:8b982703) Completed resetting hosts suitable for reconnect 2025-06-29 05:00:48,097 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-94b15a01]) (logid:8b982703) Acquiring hosts for clusters already owned by this management server 2025-06-29 05:00:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-94b15a01]) (logid:8b982703) Completed acquiring hosts for clusters already owned by this management server 2025-06-29 05:00:48,098 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-94b15a01]) (logid:8b982703) Acquiring hosts for clusters not owned by any management server 2025-06-29 05:00:48,099 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:[ctx-94b15a01]) (logid:8b982703) Completed acquiring hosts for clusters not owned by any management server 2025-06-29 05:00:48,156 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bf16283a]) (logid:cb10782d) VM scheduler.poll is being called at 2025-06-29T03:01:00+0000 2025-06-29 05:00:48,160 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bf16283a]) (logid:cb10782d) Got 0 scheduled jobs to be executed at 2025-06-29T03:01:00+0000 2025-06-29 05:00:48,161 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:[ctx-bf16283a]) (logid:cb10782d) Cleaned up 0 VM scheduled job entries 2025-06-29 05:00:48,173 DEBUG [c.c.v.V.CleanupTask] (Vm-Operations-Cleanup-1:[ctx-409b0636]) (logid:f86c2ef2) VM Operation Thread Running 2025-06-29 05:00:48,353 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-86a2c4be]) (logid:e42bc371) ClusterDRS.poll is being called at 2025-06-29T03:01:00+0000 2025-06-29 05:00:48,360 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:[ctx-86a2c4be]) (logid:e42bc371) Removed 0 old drs migration plans 2025-06-29 05:00:49,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89e43528]) (logid:d02de5d8) No inactive management server node found 2025-06-29 05:00:49,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-89e43528]) (logid:d02de5d8) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:50,432 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-8864: Seq 7-8864: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:00:50,442 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-6:[]) (logid:) SeqA 7-: Sending 8864 2025-06-29 05:00:50,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cdfe79b]) (logid:d29b4ac5) No inactive management server node found 2025-06-29 05:00:50,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1cdfe79b]) (logid:d29b4ac5) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:51,049 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-20082392]) (logid:fb4e71da) HA health check task is running... 2025-06-29 05:00:52,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e32a74a7]) (logid:171bf88d) No inactive management server node found 2025-06-29 05:00:52,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e32a74a7]) (logid:171bf88d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:53,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fa1e7a4]) (logid:8dbb949e) No inactive management server node found 2025-06-29 05:00:53,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6fa1e7a4]) (logid:8dbb949e) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:55,050 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-4d153956]) (logid:c88b8c9f) HA health check task is running... 2025-06-29 05:00:55,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5754d0d5]) (logid:d74db30a) No inactive management server node found 2025-06-29 05:00:55,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5754d0d5]) (logid:d74db30a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:56,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b24f8377]) (logid:d059be49) No inactive management server node found 2025-06-29 05:00:56,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b24f8377]) (logid:d059be49) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:58,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0b01011b]) (logid:64c486f8) Enabled non-edge zones available for scan: 1 2025-06-29 05:00:58,095 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-0b01011b]) (logid:64c486f8) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 05:00:58,096 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-0b01011b]) (logid:64c486f8) Zone [1] is ready to launch secondary storage VM. 2025-06-29 05:00:58,271 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dd9d1864]) (logid:8c20d24a) Enabled non-edge zones available for scan: 1 2025-06-29 05:00:58,276 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-dd9d1864]) (logid:8c20d24a) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 05:00:58,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45308998]) (logid:55757cef) No inactive management server node found 2025-06-29 05:00:58,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-45308998]) (logid:55757cef) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:00:59,051 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-73354e27]) (logid:0c2bba67) HA health check task is running... 2025-06-29 05:00:59,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb23624c]) (logid:d62be491) No inactive management server node found 2025-06-29 05:00:59,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb23624c]) (logid:d62be491) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:00,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-8865: Seq 7-8865: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:01:00,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) SeqA 7-: Sending 8865 2025-06-29 05:01:01,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b52f523]) (logid:3cac5494) No inactive management server node found 2025-06-29 05:01:01,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-9b52f523]) (logid:3cac5494) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:02,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c73b2ed2]) (logid:22c7cbda) No inactive management server node found 2025-06-29 05:01:02,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-c73b2ed2]) (logid:22c7cbda) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:03,052 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-02cec21a]) (logid:7e1c1fab) HA health check task is running... 2025-06-29 05:01:04,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ea3f0b4]) (logid:e808ab85) No inactive management server node found 2025-06-29 05:01:04,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-0ea3f0b4]) (logid:e808ab85) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:05,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ce5ade3]) (logid:5d48cc74) No inactive management server node found 2025-06-29 05:01:05,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ce5ade3]) (logid:5d48cc74) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:07,053 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-bccbc5e4]) (logid:9973368f) HA health check task is running... 2025-06-29 05:01:07,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f40075df]) (logid:062f8d96) No inactive management server node found 2025-06-29 05:01:07,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f40075df]) (logid:062f8d96) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:08,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f87bbd1a]) (logid:c73b60dc) No inactive management server node found 2025-06-29 05:01:08,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-f87bbd1a]) (logid:c73b60dc) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:10,361 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34ca2d4a]) (logid:131a1c03) No inactive management server node found 2025-06-29 05:01:10,361 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-34ca2d4a]) (logid:131a1c03) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:10,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-8866: Seq 7-8866: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:01:10,443 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) SeqA 7-: Sending 8866 2025-06-29 05:01:11,055 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-79a978f4]) (logid:f5326e04) HA health check task is running... 2025-06-29 05:01:11,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59e1f40f]) (logid:32ad0aff) No inactive management server node found 2025-06-29 05:01:11,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-59e1f40f]) (logid:32ad0aff) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:13,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d88deef3]) (logid:e17dc143) Found 0 routers to update status. 2025-06-29 05:01:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d88deef3]) (logid:e17dc143) Found 0 VPC's to update Redundant State. 2025-06-29 05:01:13,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-d88deef3]) (logid:e17dc143) Found 0 networks to update RvR status. 2025-06-29 05:01:13,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74f777e3]) (logid:881a4778) No inactive management server node found 2025-06-29 05:01:13,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-74f777e3]) (logid:881a4778) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:14,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1e27902]) (logid:f4c8e207) No inactive management server node found 2025-06-29 05:01:14,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d1e27902]) (logid:f4c8e207) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:15,056 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-8bb6e221]) (logid:6e329d57) HA health check task is running... 2025-06-29 05:01:15,433 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-8867: Seq 7-8867: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:01:15,442 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-8:[]) (logid:) SeqA 7-: Sending 8867 2025-06-29 05:01:16,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30563209]) (logid:b1bd1c67) No inactive management server node found 2025-06-29 05:01:16,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-30563209]) (logid:b1bd1c67) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:17,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2aa8a076]) (logid:1f2ecef3) No inactive management server node found 2025-06-29 05:01:17,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2aa8a076]) (logid:1f2ecef3) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:18,111 DEBUG [c.c.s.S.AutoScaleMonitor] (StatsCollector-1:[ctx-e8b01e43]) (logid:f7d6c61b) AutoScaling Monitor is running... 2025-06-29 05:01:18,196 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-43328d53]) (logid:fb355326) Starting to remove all volume_stats rows older than [Sat Jun 28 17:01:18 CEST 2025]. 2025-06-29 05:01:18,197 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:[ctx-43328d53]) (logid:fb355326) Removed a total of [0] volume_stats rows older than [Sat Jun 28 17:01:18 CEST 2025]. 2025-06-29 05:01:19,057 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-35a1838e]) (logid:02a03b7b) HA health check task is running... 2025-06-29 05:01:19,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c1bd645]) (logid:c2b04638) No inactive management server node found 2025-06-29 05:01:19,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1c1bd645]) (logid:c2b04638) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:19,584 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-5a739198]) (logid:098d8395) Starting to remove all vm_stats rows older than [Sat Jun 28 17:01:19 CEST 2025]. 2025-06-29 05:01:19,609 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:[ctx-5a739198]) (logid:098d8395) Removed a total of [8] vm_stats rows older than [Sat Jun 28 17:01:19 CEST 2025]. 2025-06-29 05:01:20,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5eaa9cb9]) (logid:81b2d824) No inactive management server node found 2025-06-29 05:01:20,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5eaa9cb9]) (logid:81b2d824) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:22,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb8bf97a]) (logid:7fe855d2) No inactive management server node found 2025-06-29 05:01:22,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-eb8bf97a]) (logid:7fe855d2) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:23,058 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-e42d05d5]) (logid:17f525a6) HA health check task is running... 2025-06-29 05:01:23,267 DEBUG [c.c.s.S.HostCollector] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) HostStatsCollector is running to process 2 UP hosts 2025-06-29 05:01:23,269 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 05:01:23,270 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) Seq 2-4725683383995203933: Routed from 90520738888109 2025-06-29 05:01:23,313 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) Seq 1-4725683383995203933: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 05:01:23,315 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) Wait time setting on com.cloud.agent.api.GetHostStatsCommand is 1800 seconds 2025-06-29 05:01:23,316 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) Seq 2-3926857400090362246: Routed from 90520738888109 2025-06-29 05:01:23,361 DEBUG [c.c.a.t.Request] (StatsCollector-3:[ctx-90c4e9a2]) (logid:5175c5aa) Seq 2-3926857400090362246: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2025-06-29 05:01:23,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fef788fb]) (logid:c6f45d9e) No inactive management server node found 2025-06-29 05:01:23,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fef788fb]) (logid:c6f45d9e) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:25,362 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7112361]) (logid:effc327c) No inactive management server node found 2025-06-29 05:01:25,362 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7112361]) (logid:effc327c) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:25,435 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-8868: Seq 7-8868: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:01:25,444 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-15:[]) (logid:) SeqA 7-: Sending 8868 2025-06-29 05:01:26,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b668e44]) (logid:892d513d) No inactive management server node found 2025-06-29 05:01:26,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4b668e44]) (logid:892d513d) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:27,059 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-3ec70c4c]) (logid:4a5e3062) HA health check task is running... 2025-06-29 05:01:28,088 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a81a1a39]) (logid:0d7429f1) Enabled non-edge zones available for scan: 1 2025-06-29 05:01:28,092 DEBUG [c.c.s.StatsCollector] (secstorage-1:[ctx-a81a1a39]) (logid:0d7429f1) Verifying image storage [ImageStore {"id":1,"name":"nas1","uuid":"ee21c23c-7760-47d1-9db9-d89af620656c"}]. Capacity: total=[55.8441 TB], used=[33.2330 TB], threshold=[90.00%]. 2025-06-29 05:01:28,093 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-a81a1a39]) (logid:0d7429f1) Zone [1] is ready to launch secondary storage VM. 2025-06-29 05:01:28,272 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9816be6c]) (logid:551440da) Enabled non-edge zones available for scan: 1 2025-06-29 05:01:28,278 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-9816be6c]) (logid:551440da) Zone Zone {"id": "1", "name": "az1", "uuid": "55438e7b-e5f2-4988-bbef-af0a21044016"} is ready to launch console proxy 2025-06-29 05:01:28,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a1b227d]) (logid:562d782a) No inactive management server node found 2025-06-29 05:01:28,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-1a1b227d]) (logid:562d782a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:29,860 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-252ef6cc]) (logid:4439281a) No inactive management server node found 2025-06-29 05:01:29,860 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-252ef6cc]) (logid:4439281a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms 2025-06-29 05:01:31,060 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-16935245]) (logid:9f1b1e54) HA health check task is running... 2025-06-29 05:01:31,360 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd9fde97]) (logid:fc917951) No inactive management server node found 2025-06-29 05:01:31,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-fd9fde97]) (logid:fc917951) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:31,954 DEBUG [c.c.s.S.VmStatsCollector] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) VmStatsCollector is running to process VMs across 2 UP hosts 2025-06-29 05:01:31,958 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 05:01:31,959 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) Seq 2-4725683383995203934: Routed from 90520738888109 2025-06-29 05:01:32,041 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) Seq 1-4725683383995203934: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 05:01:32,076 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds 2025-06-29 05:01:32,077 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) Seq 2-3926857400090362247: Routed from 90520738888109 2025-06-29 05:01:32,173 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-338640ce]) (logid:22cd24b4) Seq 2-3926857400090362247: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2025-06-29 05:01:32,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5487c482]) (logid:0e8ab1bd) No inactive management server node found 2025-06-29 05:01:32,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5487c482]) (logid:0e8ab1bd) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:34,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b9265fb]) (logid:ebe2858a) No inactive management server node found 2025-06-29 05:01:34,360 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b9265fb]) (logid:ebe2858a) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:34,422 DEBUG [c.c.s.S.StorageCollector] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) StorageCollector is running... 2025-06-29 05:01:34,429 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:01:34,429 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:01:34,429 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:01:34,430 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-3938116399158788188: Routed from 90520738888109 2025-06-29 05:01:34,506 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 4-3938116399158788188: Received: { Ans: , MgmtId: 90520738888109, via: 4(s-2-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:01:34,511 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:01:34,511 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:01:34,511 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:01:34,512 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-4725683383995203935: Routed from 90520738888109 2025-06-29 05:01:34,559 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 1-4725683383995203935: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:01:34,570 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:01:34,570 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:01:34,570 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:01:34,571 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-3926857400090362248: Routed from 90520738888109 2025-06-29 05:01:34,582 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-3926857400090362248: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:01:34,592 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:01:34,592 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:01:34,593 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:01:34,593 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-4725683383995203936: Routed from 90520738888109 2025-06-29 05:01:34,678 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 1-4725683383995203936: Received: { Ans: , MgmtId: 90520738888109, via: 1(cs-compute1.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:01:34,680 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand 2025-06-29 05:01:34,680 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) We are returning the default host to execute commands because the command is not of Copy type. 2025-06-29 05:01:34,680 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Wait time setting on com.cloud.agent.api.GetStorageStatsCommand is 1800 seconds 2025-06-29 05:01:34,681 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-3926857400090362249: Routed from 90520738888109 2025-06-29 05:01:34,725 DEBUG [c.c.a.t.Request] (StatsCollector-4:[ctx-22e4e275]) (logid:471803b2) Seq 2-3926857400090362249: Received: { Ans: , MgmtId: 90520738888109, via: 2(cs-compute2.cloud.redacted.net), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2025-06-29 05:01:35,061 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-1691f7ab]) (logid:cdff7e47) HA health check task is running... 2025-06-29 05:01:35,436 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-8869: Seq 7-8869: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"306","_loadInfo":"{ "connections": [], "removedSessions": [] }","wait":"0","bypassHostMaintenance":"false"}}] } 2025-06-29 05:01:35,445 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 7-: Sending 8869 2025-06-29 05:01:35,858 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-810a4ae6]) (logid:4351ed97) No inactive management server node found 2025-06-29 05:01:35,858 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-810a4ae6]) (logid:4351ed97) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:37,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ee7f4ef]) (logid:3a9c6744) No inactive management server node found 2025-06-29 05:01:37,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-4ee7f4ef]) (logid:3a9c6744) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:38,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dd464a2]) (logid:87715f06) No inactive management server node found 2025-06-29 05:01:38,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8dd464a2]) (logid:87715f06) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:39,062 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-178cfb50]) (logid:e8bf947f) HA health check task is running... 2025-06-29 05:01:40,358 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78fe7b95]) (logid:05426288) No inactive management server node found 2025-06-29 05:01:40,358 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-78fe7b95]) (logid:05426288) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:41,859 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea652d00]) (logid:a34ff90a) No inactive management server node found 2025-06-29 05:01:41,859 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-ea652d00]) (logid:a34ff90a) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:43,041 DEBUG [c.c.s.S.DbCollector] (StatsCollector-1:[ctx-6496edab]) (logid:33d1e47a) DbCollector is running... 2025-06-29 05:01:43,063 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-6:[ctx-03fae55f]) (logid:a5fc2c50) HA health check task is running... 2025-06-29 05:01:43,097 DEBUG [c.c.n.r.V.NetworkUsageTask] (RouterMonitor-1:[ctx-b4146c7f]) (logid:c94d254d) Found 0 running routers. 2025-06-29 05:01:43,098 INFO [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-5a088602]) (logid:559a89ed) Verifying the current state of [0] snapshot schedules and scheduling next jobs, if necessary. 2025-06-29 05:01:43,098 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-5a088602]) (logid:559a89ed) Snapshot scheduler is being called at [2025-06-29T03:01:43+0000]. 2025-06-29 05:01:43,099 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6cffa770]) (logid:f4352f11) Found 0 routers to update status. 2025-06-29 05:01:43,099 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:[ctx-5a088602]) (logid:559a89ed) There are [0] scheduled snapshots to be executed at [2025-06-29T03:01:43+0000]. 2025-06-29 05:01:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6cffa770]) (logid:f4352f11) Found 0 VPC's to update Redundant State. 2025-06-29 05:01:43,100 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-6cffa770]) (logid:f4352f11) Found 0 networks to update RvR status. 2025-06-29 05:01:43,117 DEBUG [c.c.n.E.ExternalDeviceNetworkUsageTask] (ExternalNetworkMonitor-1:[ctx-20fb0bbc]) (logid:7a77aca7) External devices are not used. Skipping external device usage collection 2025-06-29 05:01:43,142 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-8d084ba2]) (logid:4d106b7a) Started resource counters recalculation periodic task. 2025-06-29 05:01:43,145 DEBUG [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:[ctx-8d084ba2]) (logid:4d106b7a) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] 2025-06-29 05:01:43,306 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-994e1c1e]) (logid:be487441) Backup backup.poll is being called at 2025-06-29T03:01:43+0000 2025-06-29 05:01:43,359 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-337b41c2]) (logid:817df623) No inactive management server node found 2025-06-29 05:01:43,359 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-337b41c2]) (logid:817df623) Peer scan is finished. profiler: Done. Duration: 1ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms 2025-06-29 05:01:43,527 INFO [c.c.r.R.ResourceCountCheckTask] (ResourceCountChecker-1:[ctx-8d084ba2]) (logid:4d106b7a) Finished resource counters recalculation periodic task. 2025-06-29 05:01:44,039 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) ManagementServerCollector is running... 2025-06-29 05:01:44,040 DEBUG [c.c.u.d.DbProperties] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) DB properties were already loaded 2025-06-29 05:01:44,040 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ]. 2025-06-29 05:01:44,055 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Execution of process [4068462] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] failed. 2025-06-29 05:01:44,055 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Exit value of process [4068462] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] is [1]. 2025-06-29 05:01:44,055 WARN [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Process [4068462] for command [/bin/bash -c systemctl status cloudstack-usage | grep " Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.]. 2025-06-29 05:01:44,055 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) usage install: null 2025-06-29 05:01:44,055 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Sessions active: 0 2025-06-29 05:01:44,058 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 05:01:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068465] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ]. 2025-06-29 05:01:44,064 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c uname -r ]. 2025-06-29 05:01:44,067 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068470] for command [/bin/bash -c uname -r ]. 2025-06-29 05:01:44,068 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:01:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068471] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:01:44,073 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) system memory from /proc: 4887515136 2025-06-29 05:01:44,073 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:01:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068476] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ]. 2025-06-29 05:01:44,079 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) free memory from /proc: 359874560 2025-06-29 05:01:44,079 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 05:01:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068481] for command [/bin/bash -c ps -o rss= 4064514 ]. 2025-06-29 05:01:44,092 INFO [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) used memory from /proc: 984480 2025-06-29 05:01:44,092 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c uptime -s ]. 2025-06-29 05:01:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068482] for command [/bin/bash -c uptime -s ]. 2025-06-29 05:01:44,097 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 05:01:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068483] for command [/bin/bash -c ps -o vsz= 4064514 ]. 2025-06-29 05:01:44,108 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 05:01:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068484] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ]. 2025-06-29 05:01:44,116 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 05:01:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068491] for command [/bin/bash -c cat /proc/loadavg ]. 2025-06-29 05:01:44,120 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 05:01:44,125 DEBUG [c.c.u.s.Script] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) Successfully executed process [4068492] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ]. 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail buffersdirect.capacity, 331040 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail buffersdirect.count, 97 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail buffersdirect.used, 331040 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail buffersmapped.capacity, 0 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail buffersmapped.count, 0 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail buffersmapped.used, 0 2025-06-29 05:01:44,127 DEBUG [c.c.s.S.ManagementServerCollector] (StatsCollector-6:[ctx-3f413d3d]) (logid:cb757607) not storing detail gcPS-MarkSweep.count, 6